Parallel processing for artificial intelligence 3

Introduction This book is the third volume in an informal series of books about parallel processing for Artificial Intelligence. Like its predecessors, it is based on the assumption that the computational demands of many AI tasks can be better served by parallel architectures than by the currently popular workstations. However, no assumption is made about the kind of parallelism to be used. Transputers, Connection Machines, farms of workstations, Cellular Neural Networks, Crays, and other hardware paradigms of parallelism are used by the authors of this collection. Papers in this collection are from the areas of parallel knowledge representation, neural modeling, parallel non-monotonic reasoning, search and partitioning, constraint satisfaction, theorem proving, parallel decision trees, parallel programming languages and low-level computer vision. The final paper is an experience report about applications of massive parallelism which can be said to capture the spirit of a whole period of computing history. The articles of this book have been grouped into four chapters: Knowledge Representation, Search and Partitioning, Theorem Proving, and Miscellaneous. Most of the papers are losely based on a workshop that was held at IJCAI 1995 in Montreal, Canada. All papers have been extensively reviewed and revised resulting in a collection that gives a snapshot of the state of the art in Parallel Processing for Artificial Intelligence. In the chapter on Knowledge Representation, Shastri and Mani show how a good understanding of human neural processing can inspire and constrain the building of efficiently implemented parallel reasoning mechanisms. Boutsinas, Stamatiou and Pavlides apply parallel processing techniques to Touretzky-style nonmonotonic inheritance networks. Lee and Geller present an efficient representation of class hierarchies on a massively parallel supercomputer. Stoffel, Hendler and Saltz describe a powerful parallel implementation of their PARKA frame system. In the second chapter, all three papers, by Cook, Suttner and Berlandier and Neveu, deal with partitioning of a search space. Cook describes HyPS, a parallel hybrid search algorithm. Suttner presents static partitioning with slackness as an approach for parallelizing search algorithms. Berlandier and Neveu describe a partitioning technique applied to constraint satisfaction problems. The third chapter contains papers about parallel reasoning. Bergmann and Quantz describe a system based on the classical KL-ONE approach to knowledge representation. This system is called FLEX, as its main goal is to permit flexible reasoning. Fisher describes a technique for theorem proving that relies on the broadcast of partial results. The SiCoTHEO theorem prover of Schumann is based on competing search strategies. The last chapter contains a number of papers that do not fit well into any other category of this book. Destri and Marenzoni analyze different parallel architectures for their ability

vi to execute parallel computer vision algorithms. Kufrin's paper is about machine learning, namely the induction of a parallel decision tree from a given set of data. Lallement, Cornu, and Vialle combine methods from both connectionist and symbolic AI to build an agent-based programming language for Parallel Processing for Artificial Intelligence. Finally, Waltz presents an inside look at the development of the Connection Machine, and the problems that were arising when trying to make it financially viable. The Appendix of this book contains a list of references to papers about Parallel Processing for Artificial Intelligence that appeared at a number of workshops, giving the reader information about sources that are otherwise not easily accessible. The editors would like to thank the authors for their timely and diligent manuscript submissions, and all the reviewers for their efforts in selecting a set of high quality papers. We thank Laveen Kanal and Azriel Rosenfeld for their support in making this project happen. Additional thanks go to the staff at Elsevier, especially Y. Campfens, who patiently suffered through a sequence of delays in our preparation of the final document. We thank Y. Lee and J. Stanski of N JIT who helped with some of the "leg work" in editing the final version of this book. Finally, J. Geller would like to thank his family for giving him time off on Sundays to finish this document.

ix

About the Editors J a m e s Geller James Geller received an Electrical Engineering Diploma from the Technical University Vienna, Austria, in 1979. His M.S. degree (1984) and his Ph.D. degree (1988) in Computer Science were received from the State University of New York at Buffalo. He spent the year before his doctoral defense at the Information Sciences Institute (ISI) of USC in Los Angeles, working with their Intelligent Interfaces group. James Geller received tenure in 1993 and is currently associate professor in the Computer and Information Science Department of the New Jersey Institute of Technology, where he is also Director of the AI & OODB Laboratory. Dr. Geller has published numerous journal and conference papers in a number of areas, including knowledge representation, parallel artificial intelligence, and object-oriented databases. His current research interests concentrate on object-oriented modeling of medical vocabularies, and on massively parallel knowledge representation and reasoning. James GeUer was elected SIGART Treasurer in 1995. His Data Structures and Algorithms class is broadcast on New Jersey cable TV. Hiroaki Kitano

Dr. Hiroaki Kitano is a Senior Researcher at Sony Computer Science Laboratory. Dr. Kitano received a B.A. in Physics from International Christian University, and a Ph.D. in Computer Science from Kyoto University. He joined NEC's software Engineering Laboratory in 1984, and developed a number of very large software systems. From 1988 to 1993, he was a visiting researcher at the Center for Machine Translation, Carnegie Mellon University. In 1993, he received the Computers and Thought Award from the International Joint Conference on Artificial Intelligence. His current academic service includes, chairperson of the international committee for RoboCup (World Cup Robot Soccer), associate editor for Evolutionary Computing, Applied AI journal, and other journals, as well as executive member of various international committees. Christian Suttner

Christian Suttner studied Computer Science and Electrical Engineering at the Technische Universit~it Miinchen and the Virginia Polytechnic Institute and State University. He received a Diploma with excellence from the TU Miinchen in 1990, and since then he is working as a full-time researcher on parallel inference systems in the Automated Reasoning Research Group at the TU Miinchen. He received a Doctoral degree in Computer Science from the TUM in 1995. His current research interests include automated

theorem proving, parallelization of search-based systems, network computing, and system evaluation. Together with Geoff Sutcliffe, he created and maintains the TPTP problem library for automated theorem proving systems and designs and organizes theorem proving competitions.

Parallel Processing for Artificial Intelligence 3 J. Geller, H. Kitano and C.B. Suttner (Editors) 1997 Elsevier Science B.V.

Massively Parallel Knowledge Representation and Reasoning: Taking a Cue from the Brain Lokendra Shastri a* and D.R. Mani b aInternational Computer Science Institute 1947 Center Street, Ste. 600 Berkeley, CA 94707 bThinking Machines Corporation 14 Crosby Drive Bedford, MA 01730 Any intelligent system capable of common sense reasoning and language understanding must be capable of performing rapid inferences with reference to a large body of knowledge. The ability to perform rapid inferences with large knowledge bases is also essential for supporting flexible and effective access to the enormous body of electronically available data. Since complexity theory tells that not all inferences can be computed effectively, it is important to identify interesting classes of inference that can be performed effectively. Over the past several years we have tried to do so by working within a neurally motivated, massively parallel computational model. Our approach is motivated by the belief that situating the knowledge representation and reasoning problem within a neurally motivated computational architecture will not only enhance our understanding of the mind/brain, but it will also lead to the development of effective knowledge representation and reasoning systems implemented on existing hardware. In this chapter we substantiate this claim and review some results of pursuing this approach. These include a characterization of reflexive reasoning reasoning that can be performed effectively by neurally plausible networks; the design of CSN a connectionist semantic network that can perform inheritance and recognition in time proportional to the depth of the conceptual hierarchy; SHRUTI a connectionist knowledge representation and inference system that can encode a large number of facts, rules, and a type hierarchy, and perform a class of first-order inferences with extreme efficiency, and SHRUTI-CM5 an implementation of SHRUTI on the CM-5 that can encode over half a million rules, facts, and types and respond to reflexive queries within a few hundred milliseconds. 1. I N T R O D U C T I O N The ability to represent and use a large body of knowledge effectively is an essential characteristic of intelligent systems. For example, understanding language requires the *This work was partially funded by NSF grant IRI 88-05465, ARO grant DAA29-84-9-0027, ONR grants N00014-93-1-1149 and N00014-95-C-0182, and NSF resource grant CCR930001N.

hearer to draw inferences based on a large body of common sense knowledge in order to establish referential and causal coherence, generate expectations, and make predictions. Plausible estimates of the size of such a knowledge base range from several hundred thousand to more than a million items [8]. Nevertheless, we can understand language at the rate of several hundred words per minute. This clearly suggests that we are capable of performing a wide range of inferences with reference to a large knowledge base within a few hundred milliseconds. Any real-time language understanding system should be capable of replicating this remarkable human ability. There has been an explosive growth in electronically available information and the number of consumers of such information. The storage, transmission, access, and ultimately, the effective use of this large and heterogeneous body of data poses a number of technological challenges. A core challenge--and one that is relevant to our work--is the problem of providing intelligent content-based access to the available data. The ability to provide such access however, will depend in large part on a system's ability to bridge the "semantic gap" between a user's query and the relevant data items. This in turn would critically depend on a system's ability to perform rapid inferences based on a variety of knowledge such as ontological knowledge, terminological knowledge, domain knowledge, common sense knowledge, and user models. Several of these knowledge sources--in addition to the common sense knowledge base--will be ve.ry large and may contain several hundred thousand items. For example, the Unified Medical Language System's terminological component contains 190,863 entries consisting of medical, clinical and chemical concepts [26]. While database and information retrieval technology has evolved considerably over the past decade, the development of large-scale yet efficient knowledge based systems capable of supporting inference has lagged behind. There exist a number of robust and sophisticated database management systems that provide efficient access to ve.ry large databases, but there do not exist :high performance systems that can carry out efficient inference with respect to large knowledge bases. The integration of the inferential capabilities of an effective large-scale inference system and the full functionality of existing database and information systems should contribute to the development of a flexible, expressive, and efficient system for accessing large and heterogeneous databases. Thus from the point of view of building artificially intelligent systems capable of understanding natural language as well as from the perspective of supporting emerging technology for accessing electronically available information, it is important to develop high performance knowledge representation and reasoning systems. Complexity theo .ry however, rules out the existence of systems capable of performing all inferences effectively. Thus the key scientific challenge in building an efficient inference system consists of identif.ying interesting and useful classes of inference that can be performed effectively. AI researchers, including ourselves have pursued this goal using a number of strategies. Our approach focuses on identifying interesting and useful classes of inference that can be performed rapidly by neurally motivated and massively parallel computational models. The thesis underlying our approach is that crucial insights into the nature of knowledge representation and reasoning can be obtained by working within the computational constraints suggested by the human brain--the only extant system that exhibits the requisite attributes of response time and scalability. We believe that situating the

knowledge representation and reasoning problem within a neurally motivated computational architecture willnot only enhance our understanding of the mind/brain, but it will also lead to the development of effective knowledge representation and reasoning systems realized on existing high performance hardware platforms. In the rest of this chapter we describe some results of pursuing this approach. Section 2 discusses our approach and its motivation in more detail. Sections 3 and 4 describe two connectionist models of knowledge representation and reasoning. Section 5 describes the mapping of one of these models onto existing hardware platforms and Section 6 offers some conclusions. 2. C O M P U T A T I O N A L

EFFECTIVENESS

As the science of artificial intelligence has matured over four decades, it has become apparent that we had underestimated the complexity and intricacy of intelligent behavior. Today we realize that the task of building a system that performs intelligently in a limited domain is dramatically different from that of designing a system that displays the sort of natural intelligence we take for granted among humans and higher animals. This sharp difference is highlighted by the limitations of artificial intelligence systems developed to understand natural language, process visual information, and perform common sense reasoning. There are programs that "understand" English if the exchange is limited to talk about airplane tickets or restaurants; there are reliable vision systems that can identify a predefined set of objects presented under controlled conditions; but we have yet to design systems that can recognize objects with the skill of a monkey, or converse with the facility of a five year old. Given that existing AI systems perform credibly within restricted domains, one may be led to believe that in order to accommodate more complex domains all that is necessa .ry is to encode more facts and rules into our programs. But the situation is not so straightforward; it is not as though the existing programs are just miniature versions of larger programs that would perform intelligently in richer domains. The problem is that the solutions do not scale up: the techniques that work in restricted domains are inadequate for dealing with richer and more complex domains. As the domains grow bigger and more complex, we run into the stone wall of computational effectiveness; the performance of the system degrades and it can no longer solve interesting problems in acceptable time-scales. This is not surprising if we recognize that intelligent activity involves ve.ry dense interactions between many pieces of information, and in any system that encodes knowledge about a complex domain, these interactions can become too numerous for the system to perform effectively. A concern for computational effectiveness should be central to AI. From the viewpoint of AI, it does not suffice to offer a computational account of how an agent may solve an interesting set of problems. AI needs to solve a far more difficult problem: it must provide a computational account of how an agent may solve interesting problems in the time frame permitted by the environment. 2 The ability to satisfy the computational effectiveness constraint appears to be one of the basic properties of intelligent agents. Success, and 2The significance of computational effectiveness in the context of AI was first discussed in these terms in [17].

at times even the survival of an agent, may depend on his ability to make decisions and choose appropriate actions within a given time frame. In fact, in certain situations we would hesitate to label an activity as being "intelligent" if it takes arbitrarily long. To give an extreme example--if time were not a factor, even a dumb computer could beat the world's greatest chess player by simply enumerating the full search tree and following a path that guaranteed a win. No doubt this would take an aeon, but if time is not a factor this should not be of consequence. 3 It is tempting to ignore the computational effectiveness constraint by characterizing it as being merely a matter of efficiency or an implementation level detail. But doing so would be a mistake. Since computational effectiveness places strong constraints on how knowledge may be organized and accessed by cognitive processes, we believe that it may be essential to tackle the question of computational effectiveness at the very outset in order to understand the principles underlying the organization and use of information in intelligent systems. 2.1. C o m p u t a t i o n a l effectiveness n e c e s s i t a t e s a s t r o n g n o t i o n of t r a c t a b i l i t y As pointed out earlier, human agents perform a range of inferences while understanding language at the rate of several hundred words per minute. These inferences are performed rapidly, spontaneously and without conscious effort--as though they were a reflex response of our cognitive apparatus. In view of this we have described such reasoning as reflexive [20]. Reflexive reasoning may be contrasted with reflective reasoning which requires reflection, conscious deliberation, and often an overt consideration of alternatives and weighing of possibilities. Reflective reasoning takes longer and often requires the use of external props such as a paper and pencil. Some examples of such reasoning are solving logic puzzles, doing cryptarithmetic, or planning a vacation. W h a t should be the appropriate criteria of tractability in the context of knowledge representation and reflexive reasoning? Since polynomial time complexity is the usual "threshold" for distinguishing the tractable from the intractable in computer science, it may seem reasonable to adopt this notion of tractability in this context. But as argued in [21] reflexive reasoning requires a more stringent criterion of tractability. Let us amplify:

9 Reflexive reasoning occurs with respect to a large body of background knowledge. A serious attempt at compiling common sense knowledge suggests that our background knowledge base may contain as many as 106 items [8]. This should not be very surprising given that this knowledge includes, besides other things, our knowledge of naive physics and naive psychology; facts about ourselves, our family, friends, colleagues, history and geography; our knowledge of artifacts, sports, art, music; some basic principles of science a n d mathematics; and our models of social, civic, and political interactions. 9 Items in the background knowledge base are fairly stable and persist for a longtime once they are acquired. Hence this knowledge is best described as long-term knowledge and we will refer to this body of knowledge as the long-term knowledge base ( L T K B ) . 3Two caveats are in order. First, we are assuming that a path leading to a forced win exists, but such a path may not exist. Second, in addition to time, space or memory is also a critical resource!

9 Episodes of reflexive reasoning are triggered by "small" inputs. In the context of language understanding, an input (typically) corresponds to a sentence that would map into a small number of assertions. For example, the input "John bought a Rolls Royce" maps into just one assertion (or a few, depending on the underlying representation). The critical observation is that the size of the input, IInl, is insignificant c o m p a r e d t o the size of the long-term knowledge base ILTKBI. 4 9 The vast difference in the magnitude of ILTKBI and Ilnl becomes crucial when discussing the tractability of common sense reasoning and we h a v e t o be careful in how we measure the time and space complexity of the reasoning process. In particular, we need to analyze the complexity of reasoning in terms of ILTKB I as well as Ilnl. In view of the magnitude of ILTKBI, even a cursory analysis suggests that any inference procedure whose time complexity is quadratic or worse in ILTKBI cannot provide a plausible computational account of reflexive reasoning. A process that is polynomial in IIni however, does remain viable.

2.2. T i m e complexity of reflexive reasoning Observe that although the size of a person's ILTKBI increases considerably from, say, age five to thirty, the time taken by a person to understand natural language and draw the requisite inferences does not. This suggests that the time taken by an episode of reflexive reasoning does not depend on the ILTKBI. In view of this it is proposed t h a t a realistic criteria of tractability for reflexive reasoning is one where the time taken by an episode of reflexive reasoning is independent of ILTKBI and only depends on the depth of the derivation tree associated with the inference, s

2.3. Space complexity of reflexive reasoning The expected size of the LTKB also rules out any computational scheme whose space requirement is quadratic (or higher) in the size of the KB. For example, the brain has only about 1011 cells most of which are involved in processing of sensorimotor information. Hence even a linear space requirement is fairly generous and leaves room only for a modest constant of proportionality. In view of this, it is proposed that the admissible space requirement of a model of reflexive reasoning be no more than linear in ILTKBI. To summarize, it is proposed that as far as (reflexive) reasoning underlying language understanding is concerned, the appropriate notion of tractability is one where 9 the reasoning time is independent of [LTKB[ and is only dependent on the depth of the derivation tree associated with the inference, and

[In[ and

4A small input may, however, lead to a potentially large number of elaborate inferences. For example, the input "John bought a Rolls-Royce" may generate a number of reflexive inferences such as "John bought a car", "John owns a car", "John has a driver's license", "John is perhaps a wealthy man", etc. 5The restriction that the reasoning time be independent of ILTKBI may seem overly strong and one might argue that perhaps logarithmic time may be acceptable. Our belief that the stronger notion of effectiveness is relevant, however, is borne out by results which demonstrate that there does exists a class of reasoning that can be performed in time independent of ILTKBI.

9 the associated space requirement, i.e., the space required to encode the L T K B plus the space required to hold the working memo .ry during reasoning should be no worse than linear in ]LTKB]. 2.4. P a r a l l e l i s m The extremely tight constraint on the time available to perform reflexive inferences suggests that we must resort to massive parallelism. Many cognitive tasks, and certainly all the perceptual ones, that humans can perform in a few hundred milliseconds would require millions of instructions on a serial (von Neumann) computer, and it is apparent t h a t a serial computer will be unable to perform these tasks within an acceptable time frame [6]. The crux of the problem becomes apparent if one examines the architecture of a traditional von Neumann computer. In such a computer, the computational and the inferential power is concentrated in a single processing unit (the CPU) while the information on which the computations have to be performed is stored in an inert memo .ry which simply acts as a repository of the system's knowledge. As a result of the single processor design, only one processing step can be executed at any point in time, and during each processing step the CPU can only access a minuscule fraction of the memo .ry. Therefore, at any given instant, only an insignificant portion of the system's knowledge participates in the processing. On the other hand, intelligent behavior requires dense interactions between many pieces of information, and any computational architecture for intelligent information processing must be capable of supporting such dense interactions. It would therefore seem appropriate to treat each memo .ry cell--not as a mere reposito .ry of information, but rather as an active processing element capable of interacting with other such elements. This would result in a massively parallel computer made up of an extremely large number of simple processing elements--as many as there are memo .ry cells in a traditional computer. The processing capability of such a computer would be distributed across its memory, and consequently, such a computer would permit numerous interactions between various pieces of information to occur simultaneously. The above metaphor of computation matches the massively parallel and distributed nature of processing that occurs in the animal brain. 6 2.5. N e u r a l c o n s t r a i n t s W i t h nearly 1011 computing elements and 101'5 interconnections, the brain's capacity for encoding, communicating, and processing information is awesome and can easily support massively parallel processing. But if the brain is extremely powerfill, it is also extremely limited and imposes a number of rather strong computational constraints. First, neurons are slow computing devices. Second, they communicate relatively simple messages that can encode only a few bits of information. Hence a neuron's output cannot encode names, pointers, or complex structures. T The relative simplicity of a neuron's processing ability with reference to

~The importance of massive parallelism was discussed in the above terms in [17,18]. Several other researchers have also pointed out the significance of massive parallelism in AI. For example, see [47,11,12,32]. 7If we assume that information is encoded in the firing rate of a neuron then the amount of information that can be conveyed in a "message" would depend on AF, the range over which the firing frequency of a presynaptic neuron can vary, and AT, the window of time over which a postsynaptic neuron can "sample"

the needs of symbolic computation, and the restriction on the complexity of messages exchanged by neurons, impose strong constraints on the nature of neural representations and processes [6]. A specific limitation of neurally plausible systems is that they have difficulty representing composite structures in a dynamic fashion. Consider the representation of the fact give(John, Mary, a-Book). This fact cannot be represented dynamically by simply activating the nodes representing the roles giver, recipient, and give-object, and the constituents "John", "Mary", and "a-Book". Such a representation would suffer from cross-talk because it would be indistinguishable from the representation of give(Mary, John, a-Book). The problem is that this fact is a composite structure: it does not merely express an association between the constituents "John", "Mary", and "a-Book", rather it expresses a specific relation wherein each constituent fills a distinct role. Hence representing such a fact requires representing the appropriate bindings between roles and their fillers. It is easy to represent static (long-term) bindings using dedicated nodes and links (see Figure 1). For example, one could posit a separate "binder" node for each role-filler pair to represent role-filler bindings. Such a scheme is adequate for representing long-term knowledge because the required binder nodes may be created. This scheme however, is implausible for representing dynamic bindings arising during language understanding since these bindings have to be generated very rapidly--within a hundred milliseconds--and it is unlikely that there exist mechanisms for growth of new links within such time scales. An alternative would be to assume that interconnections between all possible pairs of roles and fillers already exist. These links normally remain "inactive" but the appropriate ones become "active" temporarily to represent dynamic bindings. This approach is also problematic because the number of all possible role-filler bindings is extremely large and will require a prohibitively large number of nodes and links. Techniques for representing role-filler bindings based on the von Neumann architecture cannot be used since they require communicating names or pointers of fillers to appropriate roles and vice versa. As mentioned above, the storage and processing capacity of nodes as well as the resolution of their outputs is not sufficient to store, process, and communicate names or pointers. As we shall see in Section 4, a t t e m p t s to solve representational problems such as the dynamic binding problem within a neurally constrained computational model lead to the identification of important constraints on the nature of reflexive reasoning.

the incident spike train. AT is essentially how long a neuron can "remember" a spike and depends on the time course of the postsynaptic potential and the ensuing changes in the membrane potential of the postsynaptic neuron. A plausible value of AF may be about 200. This means that in order to decode a message containing 2 bits of information, AT has to be about 15 msec, and to decode a 3-bit message it must be 35 about msec. One could argue that neurons may be capable of communicating more complex messages by using variations in interspike delays to encode information (e.g., see Strehler & Lestienne 1986). However, Thorpe and Imbert (1989) have argued that in the context of rapid processing, the firing rate of neurons relative to the time available to neurons to respond to their inputs implies that a presynaptic neuron can only communicate one or two spikes to a postsynaptic neuron before the latter must produce an output. Thus the information communicated in a message remains limited even if interspike delays are used as temporal codes. This does not imply that networks of neurons cannot represent and process complex structures. Clearly they can. The interesting question is how?

10

An instance

of gi

Figure 1. Static coding of bindings using binder nodes.

2.6.

Structured

connectionism

Structured Connectionist Models [6,22] are intended to emulate the information processing characteristics of the brain--albeit at an abstract computational level--and reflect its strengths and weaknesses. Arguably, the structured connectionist approach provides an appropriate framework for developing computational models that are constrained by the computational properties of the brain. Typically, a node in a connectionist network corresponds to an idealized neuron or a small ensemble of neurons, and a link corresponds to an idealized synaptic connection. The main computational features of structured connectionist models are as follows: 9 A structured connectionist model is a network of nodes and weighted links. 9 Nodes compute some simple functions of their inputs. 9 Nodes can only hold limited state information--while a node may maintain a scalar "potential", it cannot store and selectively manipulate bit strings. 9 Node outputs do not have sufficient resolution to encode symbolic names or pointers. 9 There is no central controller that instructs individual nodes to perform specific operations at each step of processing. 9 While links and link weights may change as a result of learning, they remain fixed during an episode of reflexive reasoning.

ll 2.7. M a p p i n g c o n n e c t i o n i s t m o d e l s to real machines The massively parallel structured connectionist models assume a ve.ry targe number of nodes and links, high fan-in and fan-out, and arbitrary interconnection patterns. These traits do not carry over to real machines. This shortcoming of real machines however, is offset by the fact that the processing speed and communication times of high performance platforms are several orders of magnitude faster than those assumed in connectionist models. Another important factor that facilitates the mapping of our models to real machines is the simplicity of messages exchanged by nodes. As we shall see, this allows us to leverage the active message facility provided by machines such as the Connection Machine CM-5 for low-latency interprocessor communication of short messages. Given the partial asymmetry in the strengths of connectionist models and existing hardware platforms, one needs to address several issues when mapping structured connectionist models to real machines. Some of these issues are the granularity of mapping, the coding of messages, processor allocation, and the tradeoff between load balancing and communication overhead. These issues have to be resolved based on a number of factors including the relative costs of communication, message handling, and computation, and the structural properties of the connectionist model. These issues are discussed in Section 5. 3. C S N - - A C O N N E C T I O N I S T S E M A N T I C N E T W O R K Several years ago we developed CSN, a connectionist semantic network [18] that solves a class of inheritance and recognition problems extremely fast--in time proportional to the depth of the conceptual hierarchy. In addition to offering computational effectiveness, CSN computes solutions to inheritance and recognition problems in accordance with a theory of evidential reasoning that derives from the principle of maximum entropy. The mapping between the knowledge level and the network level is precisely specified and, given a high-level specification of conceptual knowledge, a network compiler can generate the appropriate connectionist network. The solution scales because i) the time to answer questions only depends on the depth of the conceptual hierarchy, not on the size of the semantic memory, and ii) the number of nodes in the connectionist encoding is only linear in the number of concepts, properties, and property-value attachments in the underlying semantic network. Inheritance refers to the form of reasoning that leads an agent to infer property values of a concept based on the property values of its ancestors. For example, if the agent knows that "birds fly", then given that "Tweety is a bird", he may infer that "Tweety flies". Inheritance may be generalized to refer to the process of determining property values of a concept C, by looking up information directly available at C, and if such local information is not available, by looking up property values of concepts that lie above C in the conceptual hierarchy. Recognition is the dual of the inheritance problem. The recognition problem may be described as follows: "Given a description consisting of a set of properties, find a concept that best matches this description". Note that during matching all the property values of a concept may not be available locally. For this reason, recognition may be viewed as a very general form of pattern matching: one in which the target patterns are organized

12 in a hierarchy, and where matching an input pattern A with a target pattern Ti involves matching properties of A with local properties of Ti as well as with properties that Ti inherits from its ancestors. A principled treatment of inheritance and recognition is confounded by the presence of exceptions and conflicting information. Such information is bound to arise in any representation that admits default properties. Consider the following situation. An agent believes that most Quakers are pacifist and most Republicans are non-pacifist. She also knows that John is a Republican, Jack is a Quaker, and Dick is both a Quaker and a Republican. Based on her beliefs, it will be reasonable for the agent to conclude that John is, perhaps, a non-pacifist, and Jack is, perhaps, a pacifist. But what should the agent believe about Dick? Is Dick a pacifist or a non-pacifist? In [18,19] we proposed an evidential formalization of semantic networks to deal with such problematic situations. This formalization leads to a principled treatment of exceptions, multiple inheritance and conflicting information during inheritance, and the best match or partial match computation during recognition. The evidential formulation assumes that partial information about property values of concepts is available in the form of relative frequency distributions associated with some concepts. This information can bc treated as evidence during the processing of inheritance and recognition queries. Answering a query involves identifying relevant concepts and combining information (i.e., evidence) available at these concepts to compute the most likely answer. The method of estimating unknown relative frequencies using known relative frequencies is based on the principle of maximum entropy, and can be summarized as follows: If an agent does not know a relative frequency, he may estimate it by ascertaining the most likely state of the world consistent with its knowledge and use the relative frequency that holds in that world. Let us look at an informal example that illustrates the evidential approach. Consider the conceptual hierarchy shown in Figure 2 which is a generalization of the "Quaker example" mentioned above. The agent knows how the instances of some of the concepts are distributed with respect to their beliefs (pacifism or non-pacifism) and with respect to their ethnic origin (African or European). Answering an inheritance que .ry such as "Is Dick a pacifist or a non-pacifist" involves the following steps: 1. Determine the projection of the conceptual hierarchy with respect to the que .ry. The projection consists of concepts that lie above the concept mentioned in the que .ry and for which the distribution with respect to the property values mentioned in the query is known. Figure 3 shows the projected conceptual hierarchy for the example query "Is Dick a pacifist or a non-pacifist?" 2. If the projection has only one leaf, the question can be answered directly on the basis of the information available at the leaf. (In the case of our example que .ry however, the projection has two leaves Q U A K E R and R E P U B ) . 3. If the projection contains two or more leaves, combine the information available in the projection as follows: Combine information available at the leafs of the projection by moving up the projection. A common ancestor provides the reference frame for combining evidence

13 #PERSON= 200 #PERSON[has-belief,PACIFIST= 60 #PERSOM[has-belief,NON-PAC]= 140 :~RSsONN~':th~PERth'~ = 40

i

Propcrty:has-belie,f

Values:PACIFIST,NON-PAC

I

S~I

Property:has-ethEOi~O

~. ! ~ ~

I zo,~s

i

/

#MOR~ 50

I '~ #CHRIST- 60 #CHRIST[I~s-belief,PACIFISTI= 24 \#CHRIST[t~s-belief,NON-PAC]= 36

I ~_'! s" L

#QUAKER~as-belief,PACIFIST] = 7 ~_~

#QUAKE~NON-PAC] = 3

/

++ #REPUB= 80 #REPUB[has-belicf,PACIFIST]= 16 #REPUB[has-belief,NON-PAC]= 64 #REPUB[has-cth-org,AFRIC]= 5

# D E M O C = 120 #DEMOC[has-belid,PACIFIST] = 44

#REPUB[has-cth-orgJ~URO] = 75

#DEMOC[has-eth-org,EURO] = 85

#DEMOC[has-bclief,NON-PAC} = 76 #DEMOC[has-cth-org2~RIC] = 35

Figure 2. An example domain. #PERSON refer to the number of persons in the domain. #PERSON[has-belief, PACIFIST] refers to the number of persons who have the value pacifist for the property has-belief.

14

PERSON

]

................

Figure 3. The structure above the dotted line is the projection of the conceptual hierarchy (see previous Figure) that is relevant for determining whether Dick is a pacifist or a nonpacifist.

available at its descendents. The combination process is repeated until information from all the leaves in the projection is combined (at the root). In the example under discussion, the information available at QUAKER and REPUB would be combined at PERSON to produce the net evidence for Dick being a pacifist and Dick being a non-pacifist.

3.1. An evidential representation language Knowledge in the semantic memory is expressed in terms of a partially ordered set of concepts (i.e., an IS-A hierarchy of concepts) together with a partial specification of the property values of these concepts. The set of concepts is referred to as CSET, the partial ordering as <<, and the information about property values of a concept is specified using the distribution function ~;, where, 5(C, P) specifies how instances of C are distributed with respect to the values of property P. For example, 5(APPLE, has-color) may be {RED = 60, GREEN = 55, YELLOW = 23...}. Note that 5 is only a partial mapping. In terms of a traditional representation'language, knowing 5(C, P) amounts to knowing-explicitly--the values of P associated with C. For convenience we also make use of the # notation. Thus, #C[P, V] equals the number of instances of C that are observed by the agent to have the value V for property P. For example, given the above specification of 5(APPLE, has-color), #APPLE[has-color, RED] = 60. A generalization of the # notation is also used whereby #C[P1, V1][P2,V2]...[Pn,Vn] = the number of instances of C observed to have the value Vx for property P1, ... and value Vn for property pn. Observe that the representation language allows only property values to be exceptional, but not IS-A links: A concept is either an instance or subtype of another concept or it is

15 not; the << relation specifies this unequivocally. The notion of exceptions applies only to property values, and even here exceptions do not entail "'cancellation" or "blocking" of properties. This leads to a clear semantics of conceptual taxonomies while still allowing exceptional and conflicting information. In terms of the above notation, the inheritance and recognition problems may be stated as follows: I n h e r i t a n c e : Given a concept C and a property P, find I1/, the most likely value of property P for concept C. In other words, find Vi such that the most likely value of =/:/:C[P,V~] equals or exceeds the most likely value of #C[P, ~] for any other value ~ of P. R e c o g n i t i o n : Given a set of concepts, Z = {C1, C2, ...Ca}, and a description consisting of a set of property value pairs, i.e., DESCR = {[P1, V1], [P2, V2], ...[Pm, Vm]}, find a Ci such that relative to the concepts specified in Z, (7/is the most likely concept described by DESCR. In other words, find Ci such that the most likely value of: #Ci[P1, Yl], [/92, Y2], ...[Pm, Vm] exceeds the most likely value of: :#:Ci[P1, V1], [/92, V2], .-.[Pro, Vm] for any other C~- If z = {APPLE, GRAPE}, DESCR = {[has-color, RED],[hastaste, SWEET]} then the recognition problem may be paraphrased as: "Is something red in color and sweet in taste more likely to be an apple or a grape?" 3.2. C o n n e c t i o n i s t e n c o d i n g Figure 4 shows a fragment of a connectionist network that encodes some of the knowledge associated with the example shown in Figure 2. The network fragment only shows the encoding of: 9 Dick is a Quaker and a Republican. 9 Most Quakers have pacifist beliefs. 9 Most Republicans have non-pacifist beliefs. Furthermore, only nodes and links involved in computing inheritance are shown. It is assumed that one of the properties attached to persons is "has-belief", some of whose values are "pacifist" and "non-pacifist". The information about the relative frequencies of Quakers and Republicans with respect to pacifism and non-pacifism are encoded as weights on appropriate links. The encoding employs five distinct node types. These are the concept nodes, property nodes, binder nodes, relay nodes, and enable nodes. With reference to Figure 4, all solid boxes denote concept nodes, all triangular nodes denote binder nodes, and the single dashed box denotes a property node. Relay and enable nodes are not shown in Figure 4. A node is characterized by a state--either a c t i v e or inert, a potential--a real-value in the range [0,1], an output value--also in t h e r a n g e [0,1], and the functions P, Q, and V that define the values of potential, state and output at time t + 1, based on the values of potential, state and inputs at time t. For simplicity of description, we assume that nodes have multiple input sites, and incoming links are connected to specific sites (e.g., HCP, RELAY, CP, E, and EC in Figure 4). Each site has an associated site-function. These

16

BELIEF RELAY

#NON-PAC

#PACIFIST

#BELIEF

#BELIEF

#QUAKER[hb,PACI~ST]

RELAY

RELAY

NON-PAC

cP P A C I F I S T

cP

HCP

HCP #PERSON[hb.PACIFIST] #PACIFIST

PERSON RELAY

QUAKER

REPUB

RELAY

RELAY

RELAY

has-belief ....................

DICK

Figure 4. A fragment of the connectionist encoding of the example domain is depicted. The site E N A B L E has been abbreviated to E. Quantities along links denote weights. The property "has-belief" has been abbreviated to "hb" in the specification of weights. Only the weights on links from binder nodes to other concept and binder nodes pertaining to PACIFIST are shown. Analogous weights exist for NON-PACIFIST. There are links from has-belief to the E N A B L E site of each binder node in the network fragment. Only some of these links are shown.

17 functions carry out local computations based on the inputs incident at the site, and it is the result of this computation that is processed by the functions P, V and Q. The network must perform very specific computations in order to solve the inheritance and recognition problems and it must do so without the intervention of a central controller. The desired behavior is realized by introducing binder and relay nodes which mediate the spread of activation and provide a distributed control mechanism. In particular, relay nodes control the directionality of spreading activation along the IS-A links (top-down or bottom-up) while binder nodes control the flow of activation between property nodes, concept nodes, and property value nodes. The network is designed automatically by a network compiler that takes as input a high level specification of the knowledge to be encoded in the network. The syntax of the specification language parallels the representation language described above. In [18] we identify specific constraints that must be satisfied by the conceptual structure in order to achieve an efficient connectionist realization. For example, one of the constraints specifies that if the network is to correctly inherit the value of property P of a concept C, then the projection of the conceptual structure with respect to C and P must be a tree. Another constraint imposes a uniformity requirement on property value attachments in the conceptual structure and suggests that the conceptual hierarchy must comprise of several alternate "views" of the underlying concepts. A detailed discussion of these constraints is beyond the scope of this chapter.

3.3. Description of network behavior Nodes in the network can be in one of two states: active or inert. A node switches to an active state under conditions specified below and transmits an output equal to its potential. The computational characteristics of the node types shown in Figure 4 are as follows: Concept nodes: State: Node is in active state if it receives one or more inputs. Potential: If no inputs at site H C P then potential = the product of inputs at sites Q U E R Y , R E L A Y , C P and P V divided by the product of inputs at site I N V . else potential = t h e product of inputs at sites Q U E R Y , R E L A Y and H C P Binder nodes: State: Node is in active state if and only if it receives all the three inputs at site E N A B L E . Potential: If state = active then potential = product of inputs at E C else potential = 0

Property nodes switch to active state if they receive input at site Q U E R Y , and in this state their potential always equals 1.

18 3.4. T w o e x a m p l e s We illustrate the behavior of networks and the nature of inferences drawn by them with the help of two examples based on the knowledge encoded in Figure 2. Observe that there are two properties has-bel (has-belief) with values PACIFIST (pacifist) and NON-PAC (non-pacifist), and has-eth-org (ethnic-origin) with values AFRIC (African) and EURO (European). In qualitative terms, the distributional information is as follows. 9 Most persons are non-pacifist. 9 Most quakers are pacifist. 9 Most republicans are non-pacifist. 9 Most persons are of European descent. 9 Most republicans are of European descent. 9 Most persons of African descent are democrats. This information is encoded in a network in a manner analogous to that shown in Figure 4 with the distributional information encoded as weights on appropriate links. Queries to CSN are posed by activating appropriate nodes and answers are obtained by examining the level of activation (potential) of relevant nodes. The first example demonstrates how the network performs inheritance in the presence of conflicting information arising due to "multiple inheritance". Consider the inheritance query: "Is Dick a pacifist or a non-pacifist?" This question is posed to the network by activating the nodes DICK, has-belief, and BELIEF, and enabling the IS-A links from subconcepts to superconcepts. The resulting potentials of the nodes PACIFIST and NON-PAC determine whether Dick is more likely to be a pacifist or a non-pacifist. It can be shown that the potential of the node PACIFIST will equal: #QUAKER[has-bel, PACIFIST] x #REPUBLICAN[has-bel, PACIFIST] # B E L I E F x #PERSON[has-bel, PACIFIST] and that of the node NON-PAC will equal: #QUAKEa[has-bel, NON-PAC] x #REPUBLICAN[has-bel, NON-PAC] # B E L I E F x #PERSON[has-bel, NON-PAC] Ignoring the common factor, #BELIEF, in the above expressions, the potential of PACIFIST corresponds to the best estimate of the number of persons that are both quakers and republicans and believe in pacifism. Similarly, the potential of NON-PAC corresponds to the best estimate of the number of persons that are both quakers and republicans and believe in non-pacifism (for a justification, refer to [Shastri 88]). Hence, a comparison of the two potentials will give the most likely answer to the question: Is Dick a pacifist or a non-pacifist? The normalized potentials of PACIFIST and NON-PAC as a result of this query are 1.00 and 0.66 respectively. Thus, on the basis of the available information, Dick who is a republican and a quaker is more likely to be a pacifist than

19 a non-pacifist; the ratio of likelihoods being about 3:2. Similar queries for RICK, PAT, and SUSAN lead to the following results: Rick who is a Mormon republican is more likely to be a non-pacifist. The ratio of pacifist versus non-pacifist for Rick being 0.39 versus 1.00. Pat who is a Mormon democrat is also more likely to be a non-pacifist, but only marginally so (0.89 versus 1.00). Finally, Susan who is a quaker democrat is likely to be a pacifist with a ve.ry high probability (1.00 versus 0.29). As an example of recognition, consider the query: "among Dick, Rick, Susan, and Pat, who is more likely to be a pacifist of African descent?" The query is posed by activating the property nodes has-belief and has-ethnic-origin and the nodes PACIFIST and AFRIC, and enabling the top-down IS-A links. The resulting normalized potentials are for SUSAN 1.00, for PAT 0.57, for DICK 0.11, and for RICK 0.05. Susan who is a democrat and a quaker, best matches the description "person of African descent with pacifist beliefs". The least likely person turns out to be Rick. Observe that Rick is neither a democrat many of whom are of African origin, nor is he a q u a k e r - many of whom are pacifist.

4. S H R U T I - - A C O N N E C T I O N I S T

MODEL OF REFLEXIVE REASONING

Over the past years we have developed a connectionist knowledge representation and inference system, SHRUTI, that can encode a large number of facts and rules involving variables and quantifiers, and a type hierarchy, and perform a class of first-order inferences with extreme efficiency [1,23,15]. The time taken by SHRUTI t o perform an inference is just proportional to the depth of inference and is otherwise independent of ILTKB I (the size of the knowledge base). If the values of various system parameters are set to biologically plausible values, it can be shown that SHR.UTI can encode millions of rules and facts and yet draw interesting inferences in a few hundred milliseconds. Inference in SHRUTI can be viewed as the transient but systematic flow of rhythmic patterns of activity, where each phase in the rhythmic activity corresponds to a distinct entity involved in the reasoning episode and where variable bindings are represented as the synchronous firing of appropriate argument and object nodes. In SHRUTI a rule is an interconnection pattern that causes the propagation and transformation of rhythmic patterns of activity and a fact acts as a temporal pattern matcher that becomes active when it detects that the static bindings it encodes are present in the rhythmic pattern of activity. Emerging neurophysiological data suggests that the basic mechanisms proposed for representing and propagating dynamic variable bindings, namely, the propagation of rhythmic patterns of activity and the synchronous activation of nodes, exist in the brain and appear to play a role in the representation and processing of information [25]. By generalizing the results of our work on SHRUTI we have identified a class of inferences that can be performed by a parallel network whose size is linear in ILTKBI, in time that is proportional to the depth of inference and is otherwise independent of ILTKBI. This generalization relaxes the bound on the number of times a predicate may be instantiated during a derivation. Below we describe the form of knowledge that may be encoded in the network and offer a formal characterization of reflexive inference. Next we specify the computational model and give a brief overview of SHRUTI.

20 4.1. F o r m of rules, facts, a n d queries S H R U T I encodes knowledge in terms of entities, types (classes of entities), the memberof relation between entities and types, sub- and super-type relations between types, n-ary predicates, facts and rules. The sub- and super-type relations define a partial ordering of types and thus types are organized in the form of a directed acyclic graph (we will refer to it as the T-DAG). It is assumed that there exists a unique type that is a super-type of all types. For convenience we view entities as the leaves of the T-DAG. Rules have the following form: 3Xl:Xl,... , xp:Xp Vyl:Vl,..., yr:]~r [ P l ( . . . ) A ' " A Pn (. . .) =V 3Ul, . . . , ut Q ( . . .)] wherein an argument of Pi can be an entity or one of the variables xi and yi. An argument of Q can be an entity or one of the variables xi, y~, and ui. )(is and Yis are types and specify restrictions on the bindings of variables. Facts have the form: 3xl:X1,...

, x r : X r Vyl:Y1,..., Ys:Y8 P ( . . . )

where arguments of P are either entities or variables xi and Yi. Universally quantified variables are assumed to be distinct (i.e., repeated universally quantified variables are not supported). Observe that facts include ground atomic formulas as well as quantified assertions of the type "All permanent employees receive a bonus" and "there exists an employee who is the boss of all other employees". The form of queries is similar to that of facts except that whereas repeated universally quantified variables can occur in queries, existentially quantified variables are assumed to be distinct. Recently, we have extended SHRUTI to allow negative literals to occur in facts, queries, and rules [24]. The representation language shares the function-free property of Datalog [30] but differs from it in a number of ways. For example, unlike Datalog, our language does not force a dichotomy between extensional and intensional relations and allows both intensional as well as extensional relations to occur in the head of a rule. Thus the language allows rules to define new views (as in Datalog) as well as specify integrity constraints. Furthermore, the occurrence of extensional relations in the head of a rule also allows our system to reformulate a query into a number of alternate, but semantically related queries. While our language is more general than Datalog in its treatment of relations, it does impose a restriction on the form of rules (see below). 4.2. T h e class of reflexive inference A characterization of the class of reflexive inferences is provided in [21]. This characterization is facilitated by the following definitions: [] Any variable that occurs in multiple argument positions in the antecedent of a rule is a p i v o t a l variable.

[] A rule is balanced if all pivotal variables occurring in the rule also appear in its consequent. Observe that rules that do not contain pivotal variables are also balanced.

21 [] Let a derivation of a query Q obtained via backward chaining be called threaded, if all pivotal variables occurring in the derivation get bound as a consequence of bindings introduced in Q. [] Given a type hierarchy, rules, and facts, any que .ry that has a threaded derivation is called a reflexive que .ry. It can be shown that the worst case time for computing a yes answer to a reflexive yes-no que .ry Q is proportional to IInl yd, where: IInl is the number of distinct bindings specified in Q, V is the maximum arity of predicates in L T K B , and d equals the depth of the derivation of Q. Observe that V can be treated as a constant and hence, the worst case time required to answer a reflexive query is i) only proportional to d, ii) polynomial in IInl, but iii) independent of ILTKBI. An answer to a wh-que .ry can be computed in time proportional to IInl VD, except that IInl now equals the arity of the que .ry predicate Q, and D equals the greater of the two: the depth of the T-DAG and the diameter of the inferential dependency graph induced by the rules in L T K B . In [23] we had conjectured that when performing reasoning via backward chaining, any network model whose size is linear in ILTKBI and which computes inferences in time independent of ILTKBI must limit itself to answering reflexive queries involving balanced rules. A recent result [3] establishes a lower bound of f~(log n) on the time required for inferences that violate this restriction and thus provides a proof of our conjecture.

4.3. The computational model Our computational model is a network of nodes connected via weighted links. The model makes use of three node types each of which performs a simple computation and communicates via simple messages. p - b t u nodes: A p-btu node with threshold n becomes active upon receiving n synchronous inputs. In particular, a p-btu node B with threshold 1 receiving a (periodic) spike train from a p-btu node A will become active and produce a (periodic) spike train that is in-phase with the spike train it receives from A. 8 r - a n d n o d e s : A T-and node with a threshold of 1 becomes active on receiving an uninterrupted pulse of width >_ 7rmax. In other words, a T-and node behaves like a temporal and. Upon becoming active, such a node produces an output pulse similar to the input pulse. A threshold, n, associated with a T-and node indicates that the node will fire only if it receives n or more synchronous pulses. T-or n o d e : A T-Or node with threshold n becomes active on receiving n or more inputs during an interval 7r,nax. Upon becoming active, a T-Or node produces an output pulse of width 7rmax. Thus a T-or node behaves like a temporal or. The model also makes use of inhibitory modifiers that can block the flow of activation along a l i n k w a pulse propagating along an inhibitory modifier will block a synchronous pulse propagating along the link it impinges upon.

SThe "btu" in p-btu stands for binary threshold unit.

22

from John from Mary from a-Book

give

buy

from Mary from a-Ball

John

9

Mary

0

a-Book

O

s-Ball

0 csn-sell

Figure 5. An example encoding of rules and facts.

4.4. E n c o d i n g of rules a n d facts We discuss a simple example to illustrate how S H R U T I encodes rules and facts and performs inference. We suppress details pertaining to the mapping of multiple antecedent rules, the type hierarchy, and the dynamic representation of multiple instantiations of a predicate. A detailed description can be found in [23,15]. The network shown in Figure 5 encodes the following rules and facts:

1. Yx, y, z give(x,y,z) ~ own(y,z)

2. w , y buy(~,y)~ o~n(~,y)

4. give(John,Mary, a-Book) 5. 3 (x) buy(John, x) 6. own(Mary, a-Ball). The encoding makes use of two types of nodes mentioned above: p-btu nodes (depicted as circles) and v-and nodes (depicted as pentagons). The encoding of more complex rules

23

c:can-sell

f--W-V-"g f--V-W--T-W

c:own

c:give F1

e:give

I I I I I I I I I I I I

g-obj

recip 9 :buy

I I I I I I I I I I I I

b-obj buyer' e:ownl

I I I I I I I I I I I I I I

o-objl

owner e:can-sell a-Book

I I I I I I I I

=

cs-obj

input to e:can-sell ~

,.pu, to cs~jlt input to p-seller

u

I I I I I I ! 1 I I 1 1 I I I I

V

il

II

II

I I I I

I

I

Iiiiiiiii1111

I I I I I I I I I I I I I I o

1

2

3

4

5

6

7

8

9

time

Figure 6. Activation trace for the query can-sell(Mary, a-Book)?.

also makes use of the r-or nodes mentioned above. In Figure 5, inhibitory modifiers are shown as links ending in dark blobs. Each entity in the domain is encoded by a p-btu node. This acts as a "focal" node for this entity. The features of an entity and the roles it fills in various relations are encoded by linking its focal node to appropriate nodes in the network. Each n-ary predicate P is encoded by a "focal" cluster consisting of a pair of r-and nodes and n p-btu nodes, one for each of its n arguments. One of the r-and nodes is referred to as the enabler, e:P, and the other as the collector, c:P. In Figure 5 enablers point upward while collectors point downward. The enabler e:P becomes active whenever the system is being queried about P. On the other hand, the system activates the collector c:P of a predicate P whenever the system wants to assert that the current dynamic bindings of the arguments of P follow from the knowledge encoded in the system. All rules and facts pertaining to a predicate are represented by linking its focal cluster with other focal clusters and nodes as explained below. A rule is encoded by connecting the collector of the antecedent predicate to the collector of the consequent predicate, the enabler of the consequent predicate to the enabler of the antecedent predicate, and by connecting the arguments of the consequent predicate to the arguments of the antecedent predicate in accordance with the correspondence between

24 these arguments specified in the rule. A fact is encoded using a T-and node that receives an input from the enabler of the associated predicate. This input is modified by inhibitor. modifiers from the argument nodes of the associated predicate. If an argument is bound to an entity in the fact then the modifier from such an argument node is in turn modified by an inhibitory modifier from the appropriate entity node. The output of the T-and node is connected to the collector of the associated predicate. 4.5. T h e I n f e r e n c e P r o c e s s Posing a query to the system involves specifying the que .ry predicate and the argument bindings specified in the query. This is done as follows" Choose an arbitrary point in time--say, t0--as the point of reference for initiating the query (it is assumed that the system is in a quiescent state). The query predicate is specified by activating the enabler of the que .ry predicate with a pulse train of width and periodicity 7r starting at time to. The argument bindings specified in the query are communicated to the network as follows: Let the argument bindings in the query involve n distinct entities: Cl,...,c,~. With each ci, associate a delay 5i such that no two delays are within w of one another and the longest delay is less than 7 r - w. Here w is the allowable jitter (or lead/lag) between synchronously firing nodes, and 7r is the period of oscillation. Each of these delays may be viewed as a distinct phase within the period to and t0+Tr. Now the argument bindings of an entity c/are indicated to the system by providing an oscillatory spike train of periodicity 7r starting at to + (f/, to c/and all arguments to which c/is bound. This is done for each entity c/ (1 _< i < n) and amounts to representing argument bindings by the in-phase or synchronous activation of the appropriate entity and argument nodes. We illustrate the reasoning process with the help of an example. Consider the que .ry cansell(Mary, a-Book) (i.e., "Can Mary sell a-Book?") This que .ry is posed by providing inputs to the entities Mary and a-Book, the arguments p-seller, cs-obj and the enabler e:can-sell, as shown in Figure 6. Observe that Mary and p-seller receive in-phase activation and so do a-Book and cs-obj. Let us refer to the phase of activation of Mary and a-Book as Pl and p2 respectively. As a result of these inputs, Mary and p-seller fire synchronously in phase pl of every period of oscillation, while a-Book and cs-obj fire synchronously in phase P2 of every period. The node e:can-sell also fires and generates a pulse train of width 7r. The activations from the arguments p-seller and cs-obj reach the arguments owner and o-obj of the own predicate, and consequently, starting with the second period, owner and o-obj become active in pl and p2, respectively. At the same time, the activation from e:can-sell activates e:own. At this time, the system has essentially, created dynamic bindings for the arguments of predicate own and Mary has been bound to owner, and a-Book has been bound to o-obj. These newly created bindings in conjunction with the activation of e:own can be thought of as encoding the query own(Mary, a-Book) (i.e., ~"Does Ma.ry own a-Book?")! The v-and node associated with the fact own(Mary, a-Ball) does not match the que .ry and remains inactive. The activations from owner and o-obj reach the arguments recip and g-obj of give, and buyer and b-obj of buy respectively. Thus beginning with the third period, arguments recip and buyer become active in Pl, while arguments g-obj and b-obj become active in p2. In essence, the system has created new bindings for the predicates give and buy that together with the activation of the enabler nodes e:give and e:own can

25 be thought of as encoding two new queries: give(x,Mary, a-Book) (i.e., "Did someone give Maw a-Book?"), and buy(Mary, a-Book). Observe that now the T-and node associated with the fact give(John, Mary, a-Book)-this is the T-and node labeled F1 in Figure 5--becomes active as a result of the uninterrupted activation from e:give. Observe that the inhibitory inputs from recip and g-obj are blocked by the in-phase inputs from Mary and a-Book, respectively. The activation from the T-and node F1 causes c:give, the collector of give, to become active. The output from c:give in turn causes c:own to become active and transmit an output to c:can-sell. Consequently, c:can-sell, the collector of the query predicate can-sell, becomes active (refer to Figure 6) resulting in an affirmative answer to the query can-sell(Mary, a-Book)?. Conceptually, the proposed encoding of rules creates a directed inferential dependency graph: Each predicate argument is represented by a node in this graph and each rule is represented by links between nodes denoting the arguments of the antecedent and consequent predicates. In terms of this conceptualization, it should be easy to see that the evolution of the system's state of activity corresponds to a parallel breadth-first traversal of the directed inferential dependency graph. This means that i) a large number of rules can fire in parallel and ii) the time taken to generate a chain of inference is independent of the total number of rules and just equals 17r where l is the length of the chain of inference and 7r is the period of activity. The example discussed above assumed that each predicate was instantiated at most once during the inference process. In the general case, where a predicate may be instantiated several times during an episode of reasoning, the time required for propagating bindings from a consequent predicate to antecedent predicate(s) is proportional to kTr, where k is the number of dynamic instantiations of the antecedent predicate to which the bindings are being propagated.

4.6. Constraints and predictions SHRUTI identifies a number of representational and processing constraints on reflexive processing in addition to the constraints on the form of rules discussed above. These relate to the capacity of the "working memory" underlying reflexive processing and the bound on the depth of reasoning.

Working memory underlying reflexive processing: Dynamic bindings, and hence, dynamic (active) facts are represented in SHRUTI as a rhythmic pattern of activity over nodes in the LTKB network. In functional terms, this transient state of activation holds information temporarily during an episode of reflexive reasoning and corresponds to the working memory underlying reflexive reasoning (WMRR). Note that WMRR is just the state of activity of the LTKB network and not a separate buffer. Also note that the dynamic facts represented in the W M R R during an episode of reflexive reasoning should not be confused with the small number of short-term facts an agent may overtly keep track of during reflective processing and problem solving. W M R R should not be confused with the short-term memory implicated in various memory span tasks [2]. In our view, in addition to the overt working memory, there exist as many "working memories" as there are major processes in the brain since a "working memory" is nothing but the state of activity of a network. SHRUTI predicts that the capacity of W M R R is very large but at the same time it is constrained in critical ways. Most proposals characterizing the capacity of the working

26

memo~, underlying cognitive processing have not paid adequate attention to the structure of items in the working memo .ry and their role in processing. Even recent proposals such as [10] characterize working memory capacity in terms of "total activation". In contrast, the constraints on working memory capacity predicted by SHRUTI depend not on total activation but rather on the maximum number of distinct entities that can participate in dynamic bindings simultaneously, and the maximum number of (multiple) instantiations of a predicate that can be active simultaneously. B o u n d on t h e n u m b e r of d i s t i n c t entities referenced in W M R R : During an episode of reflexive reasoning, each entity involved in dynamic bindings occupies a distinct phase in the rhythmic pattern of activity. Hence the number of distinct entities that can occur as role-fillers in the dynamic facts represented in the working memory cannot exceed 7r,.~a~/w where 7rm~ is the maximum delay between two consecutive firings of cell-clusters involved in synchronous firing and ~v equals the width of the window of synchrony--i.e., the maximum allowable lead/lag between the firing of synchronous cell-clusters. If we assume that a neurally plausible value of ~rm~ is about 30 milliseconds and a conservative estimate of w is around 6 milliseconds, we are led to the following prediction: As long as the number of distinct entities referenced by the dynamic facts in the working memo .ry is five or less, there will essentially be no cross-talk among the dynamic facts. If more entities occur as role-fillers in dynamic facts, the window of synchrony w would have to shrink appropriately in order to accommodate all the entities. As w shrinks, the possibility of cross-talk between dynamic bindings would increase until eventually, the cross-talk would become excessive and disrupt the system's ability to perform systematic reasoning. The exact bound on the number of distinct entities that may fill roles in dynamic facts would depend on the largest and smallest feasible values of 7rm~ and ~v, respectively. However we can safely predict that the upper bound on the maximum number of entities participating in dynamic bindings can be no more than 10 (perhaps less). B o u n d on t h e m u l t i p l e i n s t a n t i a t i o n of r e l a t i o n s : The capacity of WMRR is also limited by the constraint that at most k dynamic facts pertaining to each relation may be active at any given time (recall that the total number of active dynamic facts can be very high). In general, the value of k need not be the same for all relations; some critical relations may have a higher value of k while some other relations may have a smaller value. The cost of maintaining multiple instantiations turns out to be significant in terms of space and time. For example, the number of nodes required to encode a rule for backward reasoning is proportional to k 2. Thus a system that can represent three dynamic instantiations of each relation may have up to nine times as many nodes as a system that can only represent one instantiation per relation. Furthermore, the worst case time required for propagating multiple instantiations of a relation also increases by a factor of k. In view of the additional space and time costs associated with multiple instantiation, and given the necessity of keeping these resources within bounds in the context of reflexive processing, we predict that the value of k during reflexive reasoning is quite small, perhaps no more than 3. B o u n d on t h e d e p t h of t h e c h a i n of reasoning: Consider the propagation of synchronous activity along a chain of role ensembles during an episode of reflexive reasoning. Two things might happen as activity propagates along the chain of role ensembles. First,

27 the lag in the firing times of successive ensembles may gradually build up due to the propagation delay introduced at each level in the chain. Second, the dispersion within each ensemble may gradually increase due to the variations in the propagation delay of links and the noise inherent in synaptic and neuronal processes. While the increased lag along successive ensembles will lead to a "phase shift", and hence, binding confusions, the increased dispersion of activity within successive ensembles will lead to a gradual loss of binding information. Increased dispersion would mean less phase specificity, and hence, more uncertainty about the role's filler. Due to the increase in dispersion along the chain of reasoning, the propagation of activity will correspond less and less to a propagation of role bindings and more and more to an associative spread of activation. For example, the propagation of activity along a chain of rules such as: 1='1(x, y, z) =v P2(x, y, z) =~ . . . Pn(x, y, z) due to a dynamic fact Pl(a, b, c) may lead to a state of activation where all one can say about Pn is this: there is an instance of Pn which involves the entities a, b, and c, but it is not clear which entity fills which role of Pn. In view of the above, it follows that the depth to which an agent may reason during reflexive reasoning is bounded. Thus an agent would be unable to make a prediction (or answer a query)--even when the prediction (or answer) logically follows from the knowledge encoded in the L T K B - - i f the length of the derivation leading to the prediction (or the answer) exceeds this bound. The actual value of this bound depends on values of appropriate physiological parameters. At this time we do not have the relevant data to arrive at a precise value, but we expect this bound to be rather low. Henderson [9] has developed an on-line parser for English using a SSRVTI-like architecture whose speed is independent of the size of the grammar and which can recover the structure of arbitrarily long sentences as long as the dynamic state required to parse the sentence does not exceed the capacity of the parser's working memory. The parser models a range of linguistic phenomena and shows that the constraints on the parser's working memory help explain several properties of human parsing involving long distance dependencies, garden path effects and our limited ability to deal with center-embedding. This suggests that the working memory constraints resulting from SHRUTI have implications for other rapid processing phenomena besides reasoning. 5. M A P P I N G

SHRUTI ONTO REAL MACHINES

Several aspects of SHRUTI suggest that a knowledge representation and reasoning system obtained by mapping SHRUTI onto real machines would be extremely efficient. These include some basic features of structured connectionism as well as constraints on rules and derivations resulting from SHRUTI. As discussed in Section 4, SHRUTI is a structured connectionist model. So it only requires nodes that perform simple computations. Second, unlike neural network models such as multilayer back-propagation networks and Hopfield nets, a SHRUTI network is sparsely connected. So even though a node may be connected to a large number of nodes, it is connected only to a relatively small percentage of nodes in the network. Consequently, only a fraction of the total number of nodes and links in the network participate in any update step. The most important source of SHRUTI'S efficiency however, is that it imposes several

28 constraints on the form of rules and derivations. These constraints were discussed in Sections 4.6 and 4.2. 5.1. M a p p i n g SHRUTI onto parallel machines In order to derive maximum benefit from the parallelism inherent in structured connectionist models, the mapping granularity must be tailored to the computational capabilities of the processors. For most real machines with relatively powerfill processors, knowledgelevel mapping provides a conceptually simple and flexible partitioning scheme. In this scheme, the knowledge base is partitioned at the relatively coarse level of knowledge elements like predicates, concepts, fact, rules and type hierarchy relations. The simplicity of the messages exchanged between nodes supports the use of interprocessor communication schemes which can handle short message packets very efficiently; complex messages and communication protocols are unnecessary. In particular, the information exchanged by nodes in SHRUTI lies in the synchronization - - o r lack t h e r e o f of converging spike trains. Given that nodes in our model are required to discriminate only among a small number of distinct phases, the necessary information can be encoded within a few bits. Consequently, information pertaining to a complete knowledge-level element can be encoded within a small message.

5.1.1. Exploiting constraints imposed by

SHRUTI

The constraints which SHRUTI imposes on the form of rules and type of inferences translates into bounds on system resources and time needed for a reasoning episode-thereby leading to a fast and efficient parallel implementation. These aspects are discussed below: 9 The form of rules and facts that can be encoded is constrained. S H R U T I attains its tractability from this fundamental constraint [21,3] which simplifies the network encoding of the knowledge base and makes it possible to perform efficient inference using spreading activation. 9 The number of distinct entities that can participate in an episode of reasoning is bounded. This restricts the number of active entities, and hence, the amount of information contained in an instantiation. In turn, this limits the amount of information that must be communicated between predicates. 9 Entities and predicates can only represent a limited number of dynamic instantiations. This constrains both space and time requirements. 9 The depth of inference is bounded. This constrains the spread of activation in the network and therefore directly affects response time and resource usage. In mapping S H R U T I onto parallel machines, we exploit these constraints to the fullest extent in order to achieve efficient resource usage and rapid responses with large knowledge bases. Of course, if any of these constraints can be relaxed without paying a severe performance penalty, we would like to obtain a more powerful system by relaxing these constraints. An example of this is the constraint on the number of instantiations of any predicate that can be active simultaneously. Based upon biological considerations, S H R U T I

29 places a limit of around 3 on this number. In our experimentation with the mapping of SHRUTI on the CM-5 we found that the limit could be raised without a major slowdown in inference times.

5.1.2. O t h e r c o n s i d e r a t i o n s The following assumptions also influenced the choices made in mapping machines:

SHRUTI

to real

9 Since the knowledge representation system should support any well-formed que .ry, the source of initial activation and the depth of derivation are unknown. In view of this, it is critical to focus on good average performance. 9 Since episodes of reasoning are expected to be rather brief, dynamic load balancing on a parallel machine is infeasible. Therefore, the static distribution of the knowledge base should be such that it leads to good dynamic load balancing on average. 9 Since the system has to reason with very large knowledge bases, the network size will be large. 5.2. SHRUTI-CM5 In this section, we briefly describe the design and implementation of S H R U T I - C M 5 , a n SPMD asynchronous message passing parallel reflexive reasoning system developed on the Connection Machines CM-5. A more detailed description of SHRUTI-CM5 can be found in [14]. 5.2.1. T h e C o n n e c t i o n M a c h i n e C M - 5 The Connection Machine model CM-5 [27] is an MIMD machine consisting of anywhere from 32 to 1024 powerful processors. 9 Each processing node is a general-purpose computer which can execute instructions autonomously and perform interprocessor communication. Each processor can have up to 32 megabytes of local memory 1~ and optional vector processing hardware. The processors constitute the leaves of a fat tree interconnection network, where the bandwidth increases as one approaches the root of the tree. A low-latency control network provides tightly coupled communications including synchronization, broadcasting, global reduction and scan operations. A high bandwidth data network provides loosely coupled interprocessor communication. The virtual machine emerging from a combination of the hardware and operating system consists of a control processor acting as a partition manager, a set of processing nodes, facilit~c~ for interprocessor communication and a CNTX-like programming interface. A typical user task consists of a process running on the partition manager and a process running on each of the processing nodes. Though the basic architecture of the CM-5 supports MIMD style programming, it is most often used to run SPMD (Single Program Multiple Data) style programs [29]. Both data parallel (SIMD) and message-passing programming on the CM-5 use the SPMD 9In principle, the CM-5 architecture can support up to 16K processors. 1~ amount of local memory is based on 4-Mbit DRAM technology and may increase as DRAMdensities increase.

30

model. If the user program takes a primarily global view of the system--with a global address space and a single thread of control--and processors run in synchrony, the operation is data parallel; if the program enforces a local, node-level view of the system and processors function asynchronously, the machine is used in a more MIMD fashion. We shall consistently use "SPMD" to be synonymous with the latter mode of operation. In this mode, all communication, synchronization and data layout are under the programs' explicit control. 5.2.2. T h e d e s i g n of SHRUTI-CM5 We outline the design and functionality of SHRUTI-CM5. A detailed discussion and justification of the design choices may be be found in [14]. T h e K n o w l e d g e Base. SHRUTI-CM5 supports all of SHRUTI'S legal rules, facts and type hierarchy relations. The type hierarchy can encode both is-a relations (which explicate the subconcept-superconcept relations between entities) and labeled relations which specify that two entities are related by a relation R. Q u e r i e s . SHRUTI-CM5 supports all the legal queries described in Section 4.1 and this includes queries posed to the rule base and/or the type hierarchy. With an appropriate front-end, the system can handle multiple queries, logical combinations of queries, and other variations. G r a n u l a r i t y of M a p p i n g . The individual processing elements on the CM-5 are fullfledged SPARC processors. A subnetwork in the connectionist model can therefore be implemented on a processor using appropriate data structures and associated procedures without necessarily mimicking the detailed behavior of individual nodes and links in the subnetwork. We therefore use knowledge-level partitioning in mapping SHRUTI onto the CM-5. This decision is also motivated by the fact that all the information pertaining to a predicate cluster can be encoded within a single CM-5 active message (see below). A c t i v e M e s s a g e s a n d C o m m u n i c a t i o n . SHRUTI-CM5 uses CMMD library functions [28] for broadcasting and synchronization, while almost all interprocessor communication is achieved using CMAML (CM Active Message Library) routines. CMAML provides efficient, low-latency interprocessor communication for short messages [28,31]. Active messages are asynchronous (non-blocking) and have very low communication overhead. A processor can send off an active message and continue processing without having to wait for the message to be delivered to its destination. When the message arrives at the destination, a handler procedure is automatically invoked to process the message. The use of active messages improves communication performance by about an order of magnitude compared with the usual send/receive protocol. The main restriction on such messages is their size--they can only carry 16 bytes of information. However, given the constraints on the number of entities involved in dynamic bindings (~ 10), there is an excellent match between the size of an active message and the amount of variable binding information that needs to be communicated between predicate instances during reasoning as specified by SHRUTI. SHRUTI-CM5 exploits this match to the fullest extent. P r o c e s s o r A l l o c a t i o n . SHRUTI-CM5 supports two major processor allocation schemes: random processor allocation and q-based processor allocation. Random processor allo-

31

while ( t e r m i n a t i o n condition not met) /* propagate a c t i v a t i o n in the type h i e r a r c h y */ spread bottom-up a c t i v a t i o n ; spread top-doma a c t i v a t i o n ; /* propagate a c t i v a t i o n in the r u l e base , / reverse-propagate c o l l e c t o r a c t i v a t i o n ; check f a c t matches; propagate enabler a c t i v a t i o n by r u l e - f i r i n g ;

Figure 7. The activation propagation loop.

cation involves allocating knowledge elements to random processors, q-based processor allocation allows the user to control the fraction of related elements that are assigned to the same processor. Random processor allocation is actually a special case of q-based allocation with q - ~1, where N is the number of processors in the parallel machine. 5.2.3. E n c o d i n g t h e k n o w l e d g e base The knowledge base is encoded by presenting rules and facts expressed in a human readable syntax like that of first-order logic. Knowledge encoding in SHRUTI-CM5 is a two-part process:

1. A serial preprocessor on a workstation reads the input knowledge base and partitions it into as many chunks as there are processors in the CM-5 partition. 2. During the parallel knowledge base encoding phase, each processor on the CM-5 independently and asynchronously reads and encodes the knowledge base fragment assigned to it by the preprocessor. This input mode parallelizes knowledge encoding and is well suited for large-scale knowledge bases. In addition, SHRUTI-CM5 also provides a direct input mode which by-passes the serial preprocessor, and is useful when small knowledge base fragments need to be added to an existing (large) knowledge base. Input processing results in allocating each knowledge base element 11 to a single processor, and encoding the knowledge using suitable internal data structures. The SHRUTI network is internally encoded by a series of pointers which serve to link predicate and concept representations. A specially designated server processor keeps track of processor assignments. The system is designed in such a manner that the server does not become a bottleneck--it is accessed only when posing a query, and does not come into play during the reasoning process. 12 11Predicates, concepts, facts, rules and is-a relations together constitute knowledge base elements. l~The server is also accessed when encoding a knowledge base in direct input mode.

20 4.1. F o r m of rules, facts, a n d queries S H R U T I encodes knowledge in terms of entities, types (classes of entities), the memberof relation between entities and types, sub- and super-type relations between types, n-ary predicates, facts and rules. The sub- and super-type relations define a partial ordering of types and thus types are organized in the form of a directed acyclic graph (we will refer to it as the T-DAG). It is assumed that there exists a unique type that is a super-type of all types. For convenience we view entities as the leaves of the T-DAG. Rules have the following form: 3Xl:Xl,... ,xp:Xp Vyl:Y1,..., yr:]~r [ P l ( . . . ) A ' " A P , ( . . . ) =V 3 u l , . . . , ut Q(...)] wherein an argument of Pi can be an entity or one of the variables xi and yi. An argument of Q can be an entity or one of the variables xi, y~, and ui. Xis and Yis are types and specify restrictions on the bindings of variables. Facts have the form: 3 x l : X l , . . . , xr:Xr Vyl:Y1,..., ys:Ys P ( . . . ) where arguments of P are either entities or variables xi and Yi. Universally quantified variables are assumed to be distinct (i.e., repeated universally quantified variables are not supported). Observe that facts include ground atomic formulas as well as quantified assertions of the type "All permanent employees receive a bonus" and "there exists an employee who is the boss of all other employees". The form of queries is similar to that of facts except that whereas repeated universally quantified variables can occur in queries, existentially quantified variables are assumed to be distinct. Recently, we have extended SHRUTI to allow negative literals to occur in facts, queries, and rules [24]. The representation language shares the function-free property of Datalog [30] but differs from it in a number of ways. For example, unlike Datalog, our language does not force a dichotomy between extensional and intensional relations and allows both intensional as well as extensional relations to occur in the head of a rule. Thus the language allows rules to define new views (as in Datalog) as well as specify integrity constraints. Furthermore, the occurrence of extensional relations in the head of a rule also allows our system to reformulate a query into a number of alternate, but semantically related queries. While our language is more general than Datalog in its treatment of relations, it does impose a restriction on the form of rules (see below). 4.2. T h e class of reflexive inference A characterization of the class of reflexive inferences is provided in [21]. This characterization is facilitated by the following definitions: [] Any variable that occurs in multiple argument positions in the antecedent of a rule is a pivotal variable. [] A rule is balanced if all pivotal variables occurring in the rule also appear in its consequent. Observe that rules that do not contain pivotal variables are also balanced.

33

QUERY DEPTH vs. RESPONSE TIME I

I

I

I

KB KB KB KB KB

0.6 spmd version 07.5 32 pe cm-5 kbl stats enabled

size size size size size

= = = = =

11.0036 219879 329871 440178 550041

', o :: :c, ;• =c

: : : ; :

0.5

"6" O

0.4

q)

E I-to

0.3

G)

0.2

0.1

0

2

4

6

8

10

Query Depth

Figure 8. SHR.UTI-CM5 query response time for artificially generated knowledge bases of different sizes.

KBSize 110036 219879 329871 440178 550041

Avg. Response Time Depth=0 Depth=l (Retrieval) 0.8 msec 1.4 msec 1.3 msec 2.5 msec 1.9 msec 4.5 msec 2.8 msec 9.2 msec 3.4 msec 15.7 msec

Table 1 Average response time for retrieval (depth 0) and depth 1 queries posed to artificially generated knowledge of different sizes.

34

!

QUERY DEPTH vs. TIME PER RULE FIRED | ,

0.0004

| KB size = 440178 KB size = 550041

spmd version 07.5 32 pe cm-5 k:bl stats enabled

0.0003

g ~0.00025

i

0.0002

!

0

2

4

6

8

Query Depth

Figure 9. Que .ry depth vs. time needed to fire a rule. Kbl on a 32 PE CM-5.

and facts, and (ii) Word Net, a real-world lexical database [16]. In this section we present these experimental results which demonstrate the effectiveness of SHRUTI-CM5 as a realtime reasoning system. Most of the experimentation has been carried out on a 32 node CM-5.

5.3.1. Experiments with artificially generated knowledge bases Part of the experimentation with SHRUTI-CM5 has been carried out using artificially generated knowledge bases. These knowledge bases are constructed automatically from a specification of their gross structure in terms of parameters such as the number of rules, facts, types and entities; the subdivision of the KB into domains; the ratio of interand intra-domain rules; and the depth of the type hierarchy. The specified structure is fleshed out with randomly generated predicates, facts, rules, and types. We have used two types of domains: target domains, which correspond to "expert" knowledge about various real-world domains; and special domains, which represent basic cognitive and perceptual knowledge about the world. A typical structured knowledge base consists of several target domains and a small number of special domains. The predicates within each domain are richly interconnected by rules. Predicates in each target domain are also richly connected by rules to predicates in the special domains. Predicates across different target domains, however, are sparsely connected. Predicates in different special domains are left unconnected. The structure imposed on the knowledge base is a gross attempt to mimic a plausible structuring of real-world knowledge bases. This is motivated by the notion that knowledge about complex domains is learned and grounded in metaphorical mappings from (to) some basic perceptually and bodily grounded domains [13]. Queries for experimenting with each artificial knowledge base were generated using the facts associated with predicates, and the inference dependency graph representing

35 rules interconnecting predicates. For the knowledge bases considered below more than 500 random queries were generated of which some 300 were answered affirmatively and had depths ranging from 0 (fact retrieval) to 8. Each successful query was run 5 times and the resulting data was used to evaluate the performance of SHRUTI-CM5. In the experimental results plotted below, points represent average values, point ranges shown are 95% confidence intervals, while the curves shown are piece-wise best-fits. Figure 8 plots response time for varying query depths and knowledge base sizes. Table 1 highlights the average response times for retrieval queries (these are queries with a derivation depth of 0) and queries with derivation depths of 1. The knowledge bases used for experimentation had 3 special domains, 150 target domains, about 50000 predicates and 50000 concepts. When the knowledge base size is about 200,000 or smaller, the response time for different query depths is essentially linear since activation is, for the most part, confined to the query domain (the domain in which the query was posed) and special domains. As the size of the knowledge base increases, the curve for each knowledge base size can be partitioned into two parts: For depths up to about 3 the response time increases steeply since all the predicates in the query domain and special domains become completely active. Beyond that, the rate at which the response time increases is lower and depends on the number of active predicates in other target domains. The latter in turn depends on the number of rules that link predicates in different target domains. As the knowledge base size increases, the number of inter-domain rules increases, and hence, the response time increases at a higher rate. This is illustrated by the top three curves in the figure. Figure 9 shows the time needed to fire a rule as a function of query depth for two knowledge bases. In general, it was found that for large knowledge bases and que .ry depths greater than 3, the number of rule firings per second on a 32 node CM-5 converged to a relatively constant value of about 125,000. This suggested that the number of rule firings per second per processor was 125.,000 52 , i.e., about 3,900. In other words, T, the time per rule firing per processor on a 32 node CM-5 was found to be about 39-~, i.e., 256 #sec. Experimental results showed that this value of T remained constant over various knowledge base structures and numbers of CM-5 processors. This means that if ]LTKB I is the size of the knowledge base, and a fraction r of this knowledge base becomes active during a reasoning episode, then the expected response time of SHRUTI-CM5 o n a n N processor CM-5 would be T = ILTrRIrT N The above observation suggests a way of designing a real-time reasoning system. Let T,nax be t h e maximum response time that the application can tolerate. The observation suggests that one requires:

Tmax <_ k]LTKB]rT N where k is a factor of safety whose value depends on the severity of the penalty for a response time exceeding Tmax. Based on this equation, we can estimate the largest knowledge base that will provide real-time responses on a given CM-5 partition. Conversely, given a knowledge base, we can determine the size of the CM-5 partition needed in order to obtain real-time performance. This assumes that (i) the dominant component of computation costs is the processing of rules and (ii) the computation is uniformly distributed

35

KB Size 329871 440201 550049

Average Speedup 64 P E CM-5 128 P E CM-5 1.82 2.96 1.67 3.10 1.76 3.39

Table 2 Average speedup for various knowledge base sizes. The maximum speedup possible for 64 and 128 node machines over a 32 node machine is 2 and 4, respectively.

SHRUTI-CM5 Query is-a hypernym (Bird,x)? is-a hyponym (Sparrow,x)? is-a synonym (Bird,x)? is-a hypernym (Entity, Sparrow)?

Response Time 0.029 sec 0.007 sec 0.004 sec 0.275 sec

Serial WordNet Query wn bird-treen wn sparrow -hypen wn bird-synsn -

Response Time 13.27 sec 0.046 sec 0.023 sec -

SHRUTI-CM5 Speedup 457 6.5 5.75 -

Figure 10. Timing data for some WordNet queries.

on all the processors of the machine. We have also experimented with artificially generated knowledge bases on 64 and 128 node CM-5 machines. Table 2 shows average speedups obtained (averaging over all que .ry depths) for a given knowledge base size as compared to the performance on a 32 node CM5. This speedup in suggests that SHRUTI-CM5 does exploit parallelism and can benefit from the availability of even greater parallelism.

Pile .'Udinlalle Pl~pm'Cin .................................................................................

o i-INNI--IN .

N

N ~ N N D D

~

FOe

~.o 3~7.1S

~

,~ E ] D D S D D

,~ E ] D N D N D

,, [Z] I-11-1 [3 D I-] ~. I-I V11-I I-I D [-1 "P~0CES~QR COHPUTATION~LLOAD FOR QUERY is-a

.4Jdmmal~ P i ~ s

~J~.6

,0 V ] D D D ~ N ~. D D V 1 N D D

(Enttt . S p a r ~ ) ? C~l= 0.03)"

~

1

)

1

LOI~DFOR QIJEIRuIs.-a

lr

(Erlt.n . S ~ 7

(

: 0.03)"

Figure 11. CM-5 processor computation (left) and communication (right) load for WordNet query is-a hypernym (Entity, Sparrow)?

37 5.3.2. Experiments with W o r d N e t WordNet is an on-line lexical database which attempts to organize information in terms of word meanings and their inter-relationships [16]. Groups of synonymous words, termed synsets, are used to represent meanings. The meanings of synsets are then defined in terms of their relationships with other synsets. For example, the ent .ry {Cock, Rooster, Chicken, @} has two words, Cock and Rooster, in the synset; this synset is a hyponym (subconcept) of the Chicken synset. WordNet (version 1.4) has about 75,000 synsets classified into nouns, verbs, adjectives and adverbs. WordNet has been mapped to SHRUTI by representing: (i) word-senses and word-forms by nodes; (ii) synsets by a pair of "focal" nodes connected to the appropriate word-sense and word-form nodes, and (iii) lexical relations between words (like antonym) and semantic relations between synsets (like hypernym or superconcept) as labeled is-a relations. The entire database translates into about 880,000 is-a relations. SHRUTI-CM5 supports labeled is-a relations of the form is-a R (A,B), which represents A _+R B. For example, is-a hypernym (Bird, Sparrow) asserts that Bird is a hypernym (or superconcept) of Sparrow. The system supports both specific queries like is-a hypernym (Bird, Sparrow)? ("Is Bird a hypernym of Sparrow?") and enumeration queries like is-a hypernym (Bird, x)? ("Enumerate entities which have Bird as a hypernym"). Even though WordNet exercises only the type hierarchy of SHRUTI-CM5, the similarity of activation propagation in the type hierarchy and rule base ensures that WordNet helps evaluate the effectiveness of SHRUTI-CM5 in terms of its ability to handle large real-world knowledge bases. Figure 10 shows the performance of WordNet on SHRUTI-CM5. For enumeration queries, the figure also compares the performance of SHRUTI-CM5 with the serial command line interface to WordNet. 13 The times shown offer a fair comparison and only take into account the time required to compute the results- they do not include the time required to display or output the results. The timing reported for the serial version is the minimum obtained in six consecutive runs of the same query (to avoid counting time to process cache misses) on a Sun Sparc Station 10. SHRUTI-CM5 timings are averages over ten runs. Figure 11 shows the computation and communication load distribution--i.e., the number of active entities and the number of messages sent and received, respectively--on the CM-5 processors for the query is-a hypernym (Entity, Sparrow)? For a detailed description see [14]. 6. C O N C L U S I O N In this chapter we have shown that there exist interesting and useful classes of inference that can be performed rapidly by neurally motivated and massively parallel computational models. These include a class of inheritance and recognition problems and a class of reflexive first-order inferences. We have also shown that these models can be mapped effectively onto existing hardware platforms and result in a high performance knowledgebased inference system. We believe that the effectiveness of SHRUTI-CM5 validates our belief that situating the knowledge representation and reasoning problem within a neurally motivated computational architecture may not only enhance our understanding of the mind/brain, but it 13We have modified the serial WordNet interface wn to report computation time using Unix timers.

38 may also lead to the development of effective knowledge representation and reasoning systems implemented on existing hardware platforms. In ongoing work we are extending the inferential power of SHRUTI-CM5 by systematically relaxing some of the biologically motivated constraints suggested by SHRUTI. The realization of S H R U T I on real machines offers an interesting set of trade-offs and we are investigating these in order to design a system that can respond rapidly to reflexive queries and at the same time perform reflective inferences at a slower speed. We are also investigating the possibility of integrating the inferential capabilities of SHRUTI-CM5 with the full functionality of existing database and information systems in order to develop a flexible and efficient mediator for accessing large and heterogeneous databases. REFERENCES

1. V. Ajjanagadde and L. Shastri. Rules and variables in neural nets. Neural Computation, 3:121-134, 1991. 2. A. Baddley. Working Memory. Clarendon Press, 1986. 3. P. Dietz, D. Krizanc, S. Rajasekaran, and L. Shastri. A lower bound result for the common element problem and its ,implication for reflexive reasoning. Technical Report MS-CIS-93-73, Department of Computer and Information Science, University of Pennsylvania, 1993. 4. M.P. Evett, W. A. Andersen, and J. A. Hendler. Providing computationally effective knowledge representation via massive parallelism. In L. Kanal and V. Kumar, editors, Parallel Processing for Artificial Intelligence, New York, 1994. Elsevier Science Publication. 5. S.E. Fahlman. NETL: A System for Representing and Using Real World Knowledge. MIT Press, Cambridge MA, 1979. 6. J.A. Feldman and D. H. Ballard. Connectionist models and their properties. Cognitive Science, 6(3):205-254, 1982. 7. J. Geller and C. Du. Parallel implementation of a class reasoner. Journal of Theoretical Artificial Intelligence, 3:109-127, 1991. 8. R . V . Guha and D. B. Lenat. CYC: A mid-term report. AI Magazine, 11(3):32-59, 1990. 9. J. Henderson. Connectionist syntactic parsing using temporal variable binding. Journal of Psycholinguistic Research, 23(5):353-379, 1004. 10. M. A. Just and P. A. Carpenter. A capacity theory of comprehension: Individual differences in working memory. Psychological Review, 99(1):122-149, 1993. 11. H. Kitano. Challenges of massive parallelism. In Proceedings of the Thirteenth International Joint Conference on Artificial Intelligence, pages 813-834, 1993. 12. H. Kitano, H. Hendler, T. Higuchi, D. Moldovan, and D. Waltz. Massively parallel artificial intelligence. In Proceedings of the Thirteenth International Joint Conference on Artificial Intelligence, pages 557-562, 1991. 13. G. Lakoff and M. Johnson. Metaphors We Live By. University of Chicago Press, Chicago, 1980. 14. D. R. Mani. The Design and Implementation of Massively Parallel Knowledge Representation and Reasoning Systems: A Connectionist Approach. PhD thesis, Depart-

39 ment of Computer and Information Science, University of Pennsylvania, 1995. 15. D. R. Mani and L. Shastri. Reflexive reasoning with multiple instantiation in a connectionist reasoning system with a type hierarchy. Connection Science, 5(3 & 4):205-242, 1993. 16. G. A. Miller, R. Beckwith, C. Fellbaum, D. Gross, K. Miller, and R. Tengi. Five papers on WordNet. Technical Report CSL-43, Princeton University, July 1990. Revised March 1993. 17. L. Shastri. Massive parallelism in artificial intelligence. Technical Report MS-CIS86-77, University of Pennsylvania, 1986. 18. L. Shastri. Semantic Networks: An Evidential Formulation and its Connectionist Realization. Morgan Kaufmann, San Mateo, CA, 1988. 19. L. Shastri. Default reasoning in semantic networks: A formalization of recognition and inheritance. Artificial Intelligence, 39(3):283-355, 1989. 20. L. Shastri. The relevance of connectionism to AI: A representation and reasoning perspective. In J. A. Barnden and J. B. Pollack, editors, Advances in Connectionist and Neural Computation Theory, Volume 1. Ablex Publishing Corporation, Norwood, N J, 1991. 21. L. Shastri. A computational model of tractable reasoning--taking inspiration from cognition. In Proceedings of the Thirteenth International Joint Conference on Artificial Intelligence, 1993. 22. L. Shastri. Structured connectionist models. In M. A. Arbib, editor, The Handbook of Brain Theory and Neural Networks. MIT Press, 1995. 23. L. Shastri and V. Ajjanagadde. From simple associations to systematic reasoning: A connectionist representation of rules, variables and dynamic bindings using temporal synchrony. Behavioral and Brain Sciences, 16(3):417-494, 1993. 24. L. Shastri and D. J. Grannes. Dealing with negated knowledge and inconsistency in a neurally motivated model of memory and reflexive reasoning. Technical Report TR-95-041, International Computer Science Institute, 1995. 25. W. Singer. Synchronization of cortical activity and is putative role in information processing and learning. Annual Review of Psysiology, 55:349-74, 1993. 26. U. K. Sources, April 1994. National Library of Medicine. 27. TMC. Connection machine CM-5 technical summary. Technical Report CMD-TS5, Thinking Machines Corporation, Cambridge, MA, 1991. 28. TMC. CMMD Reference Manual. Version 3.0. Thinking Machines Corporation, Cambridge, MA, 1993. 29. TMC. CM-5 User's Guide. CMost Version 7.3. Thinking Machines Corporation, Cambridge, MA, 1994. 30. J. D. Ullman. Principles of Database and Knowledge-Bases Systems. Computer Science Press, 1988. 31. T. yon Eicken, D. E. Culler, S. C. Goldstein, and K. E. Schauser. Active messages: A mechanism for integrated communication and computation. In Proceedings of the Nineteenth International Symposium on Computer Architecture. ACM Press, 1992. 32. D. L. Waltz. Memou-based reasoning. In M. A. Arbib and J. A. Robinson, editors, Natural and Artificial Parallel Computation, pages 251-276. MIT Press, Cambridge, MA, 1990.

40 L o k e n d r a Shastri

Lokendra Shastri is a Member of the AI Group at the International Computer Science Institute (ICSI) Berkeley, CA. His research interests lie at the intersection of Artificial Intelligence, Cognitive Science and Neural Computation. In particular, he is interested in the role of massive parallelism in knowledge representation and reasoning; neurally motivated models of reflexive (i.e., rapid) reasoning and one-shot learning of events and situations; and spatio-temporal networks for pattern recognition. Shastri is on the editorial board of Connection Science and IEEE Expert. Before joining ICSI Berkeley, he was on the faculty of the University of Pennsylvania. He received a Bachelor of Engineering degree in Electronics with distinction from the Birla Institute of Technology and Science, India, an M.S. in Computer Science from the Indian Institute of Technology, Madras, and a Ph.D. in Computer Science from the University of Rochester in 1985. D. R. M a n i

D. R. Mani is a principal research engineer at Thinking Machines Corporation, Bedford, MA. His research interests span the application of massively parallel processing to artificial intelligence, knowledge representation, and database systems. He is currently involved in the design and development of parallel machine learning algorithms for largescale data mining and knowledge discovery systems. Mani has a Bachelor's degree in Electronics and Telecommunication Engineering from the University Visvesvaraya College of Engineering, Bangalore, India, a Master's degree in Computer Science from the Indian Institute of Technology, Kanpur and a Ph.D. in Computer Science from the University of Pennsylvania, Philadelphia, PA.

Parallel Processing for Artificial Intelligence 3 J. Geller, H. Kitano and C.B. Suttner (Editors) 1997 Elsevier Science B.V.

41

Massively Parallel Support for Nonmonotonic Reasoning B. Boutsinas, a Y. C. Stamatiou, ~* and G. Pavlides ~ aDepartment of Computer Engineering and Informatics, University of Patras, 26500, Rio, Patras, Greece This work presents how Weighted Inheritance Networks, a new formalism that is capable of representing knowledge under a nonmonotonic multiple inheritance scheme with exceptions, support the design of massively parallel algorithms for problems related to nonmonotonic reasoning. Weighted Inheritance Networks are semantic networks with links that are annotated with symbolic pairs that we call weights. They can handle the inheritance structures described in the literature, including some which the so far proposed formalisms may fail to treat satisfactorily. Weighted Inheritance Networks support the development of both sequential and parallel efficient algorithms, without sacrificing any desired properties that inheritance networks should, in general, possess. We investigate the performance of parallel nonmonotonic multiple inheritance reasoners based on Weighted Inheritance Networks, for two important problems: the goal-directed inheritance reasoning problem, that consists in answering whether an object possesses a particular property and the recognition problem, that consists in finding all the objects possessing a particular set of properties. We present a system that provides massively parallel support for nonmonotonic "path-based" reasoning, called WINBRID, and we analyze its performance using randomly generated knowledge bases. 1. I N T R O D U C T I O N Nonmonotonic reasoning plays an important role in the development of systems that try to mimic commonsense reasoning. Human beings are constantly forced to make decisions and reach conclusions in a fuzzy world. The knowledge that can be acquired by observation is inherently incomplete and may even contain conflicting information as well as exceptions to general rules. Formal logic systems do not seem to provide a satisfactory solution in such an environment (see for example [19,3]) in terms of efficiency. It has been shown, that many of these systems have decision problems that are intractable, or even undecidable, when stated within their context. The algorithmic intractability of the proposed formal systems, along with their weak treatment of incomplete or contradictory knowledge, have led to the use of semantic networks in order to represent knowledge. The inheritance systems that are based on such a representation are capable of nonmonotonic multiple inheritance reasoning. Their semantics are supplied either indirectly, through a translation *This research was partially supported by the European Union ESPRIT Basic Research Projects ALCOM II (contract no. 7141), GEPPCOM (contract no. 9072) and Insight II (contract no. 6019).

42 into an extension of a standard logical formalism (see for example [5]), or directly, through "path-based" theories (see for example [21]). In this work, we propose a form of semantic networks with annotated links called Weighted Inheritance Networks (first defined in [2]), or WINs for short, as an alternative "path-based" approach that is capable of nonmonotonic multiple inheritance reasoning with exceptions. We also present how WINs treat the inheritance structures described in the literature, including those that are supplemented with "redundant generic statements" (see section 3), which the systems proposed so far may fail to treat satisfactorily ([12]). There is a clash of intuitions concerning the treatment of nonmonotonicity (on-path versus off-path preemption), the treatment of competing extensions (forms of skepticism versus forms of credulity) and the direction in which the represented inheritance is followed (upward versus downward view of inheritance) in many "path-based" inheritance systems (see [22]) that have appeared in the literature. It should be stressed that the way in which nonmonotonicity and multiple extensions are treated, is the principal cause for this state of affairs and the confusion that may result thereof. In this work, we give a formal definition of WINs, with these intuitions in mind, and we examine their semantics and algorithmic properties, focusing on the support they can offer for developing massively parallel, multiple inheritance reasoners. Then we examine the support that WINs offer for efficient parallelism when confronted with two well-known problems: the goal-directed inheritance reasoning problem, that consists in answering whether an object possesses a particular property and the recognition problem, that consists in finding all the objects possessing a particular set of properties. Developing efficient algorithms for these two problems is of great importance for many AI applications. A WIN could be, for example, a part of an inference engine of a knowledge-based system or a guiding formalism for structuring the case memory of a learning apprentice that incorporates a case-based learning mechanism. In that latter case, an efficient parallel solution to the recognition problem is highly desirable. This is the case, because case-based systems resort to a knowledge-base (usually of a large size) containing previously stored problems and solution/failure plans in order to recall similar problem situations. Recognition is therefore a frequent operation that, if implemented inefficiently, has a dramatic impact on the performance of the system [15]. WINs support the design of NC algorithms for the above problems. The class NC is the class of problems that are amenable to efficient parallel solutions, that is, solutions that use a polynomial number of processors and take time polynomial in the logarithm of the input size (see [14]). Problems that belong to this class are efficiently parallelizable. If we restrict the number of exceptions to be of the order of log n, where n is a measure of the size of the knowledge base, we obtain NC algorithms for the problems stated above. To the best of our knowledge, no NC algorithm has been given for these problems under other inheritance network formalisms. More specifically, for a reasoner based on a WIN whose underlying graph has n nodes and e edges, we present a parallel algorithm that solves the goal-directed inheritance reasoning problem in O(log 2 n) time, using O(2keM(n)) processors, where k is the maximum number of nodes that represent classes/properties that are cancelled by exceptions, and M(n) is the number of processors needed to multiply two n x n matrices over the ring (Z, +, x) in time O(log n). At the time of this writing, i ( n ) = O(n 2"376) (see [4]). Our model of parallel computation is the Concurrent Read Concurrent Write (CRCW) PRAM

43

shared memory machine (see [13] for a detailed description of this model). Additionally, a hybrid algorithm is also presented (a hybrid algorithm is a sequential one that includes some steps that are executed in parallel) with a time complexity determined by the factor n + e using O(n) processors. As far as the recognition problem is concerned, given a set of p features that must be satisfied and a set of objects, an O(log 2 n) time, O(2kpM(n)) processor, parallel algorithm for the same model of computation is presented. Also, a hybrid algorithm is given for this problem with the same performance as the algorithm for the previous problem. WINBRID (Weighted Inheritance Network hyBRID) is a reasoner capable of nonmonotonic, multiple inheritance reasoning that is based on WINs, that uses the hybrid algorithms in order to handle the two problems mentioned above. It is implemented under the PARIX environment for software development, in Parsytec's GCel-3/512 parallel machine. In what follows, we first present WINs, then we present the algorithms and finally we discuss the architecture of WINBRID.

2. W E I G H T E D

INHERITANCE

NETWORKS

The underlying structure of a Weighted Inheritance Network is a directed acyclic graph. Knowledge is represented by attaching to each node of the graph a label that denotes an object, a class of objects or a property possessed by objects and by establishing the desired inheritance structure through the insertion of properly directed edges. When we insert an edge that emanates from a node and is directed to another, we mean that the class of objects represented by the former node inherits some or all of the defining properties of the class represented by the latter. We believe that such a notion of inheritance is based on the reusability notion. For instance, when human beings think of the class/concept Whale, they first think that a whale "is a fish", because it really looks like a big fish, and then they add, as an afterthought, that it "is a mammal". This is an example of the priority that is apparently given to resemblance over class inclusion. This priority leads us to think that reusability must be an important ingredient in the process through which a human being reaches the above decision, since in order to define the class Whale there are more characteristic parts we can reuse from the definition of the class Fish than from the definition of the average member of the class Mammal. The precedence of resemblance over inheritance is not obvious, since objects (classes or individuals) usually reuse the definition of the class they belong to and, hence, there is the impression that inheritance is based principally on subclass inclusion. Thus, inheritance relates the various classes according to the possibility of reusing all or part of the definition of one in the definition of the other. As an example, if we have defined the class Elephant we can reuse this definition in order to define the object Clyde, the elephant. Also, if we have defined both Republican and Quaker, we can reuse part of each of these definitions to define the object Nixon, the Republican and Quaker. Which part we reuse depends on how the preference over possible conflicting choices is established. To be consistent with our belief that inheritance is based on the reusability notion, we will substitute the usual "isa" ("nisa") hierarchy by the "isl" ("nisl") hierarchy, which stands for "is like" ("is not like"). If an exception exists in reusing the definition of a class in the definition of an object

44 (class or individual), this is indicated by a symbolic pair that we call weight attached to the edge(s) emanating from the node which represents this object and belonging to paths which lead to the node that represents this class. A weight w has the form (s, c) and represents the possibility or the impossibility, depending on whether s = '+' or s = ' - ' , respectively, of reusing the definition of the class c in the definition of the entity from which the edge labeled by w emanates. This possibility or impossibility has higher priority than what can be normally derived by following the edges. In the proposed systems that are capable of representing knowledge under a multiple inheritance scheme with exceptions (e.g., [17,21,11]), if there exists an exception in reusing a definition, this is indicated by an exception link. However, the exception links no longer represent uniformly the reusability notion mentioned above. Instead, these links represent cancellations of the facts derived by the normal process of reusing definitions. Intuitively, this lack of uniformity is the reason why parallel inference algorithms, especially for multiple extensions, have severe limitations (see [21,6,16,12]).

~ ~ ~ --

Clyde

%A

x~ 1.

Clyde

9

-

-

"

RoyalElephant (-,Gray)

/ - -

RoyalElephant

~

o

~

~

~

1 [ .

%-.

Elephant

~

~

9

~

i ~

\~j,.

Gray

x-

x-

/ I .

/ 1 .

Elephant

Gray

Figure 1. An Inheritance Network and a Weighted Inheritance Network

Figure 1 illustrates an example of an inheritance structure that introduces nonmonotonicity. This example is:

isl(Clyde, RoyaIElephant)and isl(RoyaIElephant, Elephant)and isl(Elephant, Gray) and nisl(RoyalElephant, Gray) Here we should be able, of course, to infer nisl(Clyde, Gray). This is achieved by the existence of the weight (-,Gray) along the edge (RoyalElephant, Elephant). This weight is propagated along the path RoyalElephant ~ Elephant ~ Gray, thus excluding "Gray" from the definition of "Clyde", since the exception takes precedence over following the hierarchy links towards the node labeled "Gray". Generally, objects (individuals or classes) are considered to possess a set of properties or characteristics. Therefore, in order to define an object, all of its characteristics should be collected. This is achieved by following all the directed paths passing through the object under consideration, while gathering information about how this object looks like (that

45 is, about the possibility of reusing the characteristics of another class), thus collecting the actual characteristics t h a t it has. This information is represented by both the nodes a n d the weights of a WIN. We denote by C S the set of characteristics of an object that are collected by a reasoner in this way. This set is derived from two other ordered sets: the ordered set LS, whose elements are obtained by following the "looks-like" links and the ordered set W S that consists of the weights that are attached to those links. Notice that new elements are placed at the ends of the sets W S and LS. The elements of W S take precedence over the elements of L S and are ordered in decreasing priority, in the sense that an element has higher priority than its successor in the order they were gathered following the directed paths, according to Touretzky's inferential distance ordering. Therefore, if there are any conflicts among the elements of W S , they are resolved in favor of the element that appears earlier. By removing one of the conflicting elements, taking into account their relative precedences, we obtain the W S - set, the set of relevant weights. The set of characteristics of an object C S is the W S - set augmented, generally, by elements of L S which are not the negation of elements of W S - .

2.1. Formal definitions Before we give the formal exposition of the ideas discussed in the previous section, we will provide some preliminary notations and assumptions. Definition 1 (Weighted Inheritance N e t w o r k ) A Weighted Inheritance Network is an ordered pair W - (G(V, E), ~') where G(V, E) is a directed acyclic graph with V the set of nodes and E the set of edges and jc is a mapping from E to sets of weights and it is specified by the designer of the knowledge base. The set V (the objects) is partitioned into two sets, the set I (the individual entities) and the set C (classes or concepts). The set E is a set of ordered pairs (o, c) where o E V and c C C. The mapping ~ : E ~ 2 H associates each edge with a subset of the set of weights H where H is a set of ordered pairs of the form (s, n) where s E { + , - } and n e C. Since the graph G is acyclic, there is a function f : V ~ N that assigns to each node an integer, such that if node i is an ancestor of node j (in the sense that there exists a directed path from i to j in G), then f ( i ) < f ( j ) (topological ordering). Therefore, if we are given some nodes that lie on the same path, then f induces a total order on them. We observe that if we assign to each edge e -- (u, v) the number that is assigned to u, then a total order (one edge precedes another) is also induced on the edges that lie on the path. In what follows, by S[i] we denote the ith element of the ordered set S.

Definition 2 (inheritance p a t h ) An inheritance path p (from now on called path) from node s to node t, denoted by s ~ t, is a sequence of nodes of the form s = el, e2, . . . , ei = t, i > 1, such that (ei, ej+l) E E. Since G is acyclic, the nodes e l , . . . , ei are distinct. The length of the path, denoted by IPl, is equal to i - 1. Moreover, we require the following important property: Vs, t E V, if p is a path s ~ t : IPl > 1, then (s,t) fg E This property will be referred to as the non-shortcut property and simply states that there is no path whose endpoints are "short-circuited" by an edge. This property can be easily imposed during the construction or the update of a WIN in a preprocessing phase.

45 Definition 3 (looks-like hierarchy) Given an object o E V and a path p that starts from o, we define the set of characteristics of o according to the "is-like" hierarchy to be the ordered (according to f ) set LSo,p = {(+,j)lonpath(j,p)}, where onpath(j,p) is true if the node j belongs to p. For instance, in the example of figure 1, we obtain the following set: n SClyde,C lyde--+Royal Elephant-~ Elephant--+Gray --

{ (+, Clyde), (+, RoyalElephant), (+, Elephant), (+, Gray) } Definition 4 (the set of weights) Given an object o C V, and a path p that starts from o, we define the set of weights to be the ordered set WSo,p - {U(~,.i)eE~((i,j))ionpath(i,p)A onpath(j,p)}. Each element of WSo,p (which is a set of weights), possesses the same order as the edge to which it belongs.

For instance, in the example of figure 1, we obtain the following set: WSClyde,Clyde-+noyalElephant--+ Elephant--+Gray z { (--, Gray)}

Definition 5 (negation of a weight) If x = (s, c) is a weight, then -~x = (s', c) where s'

e

{-, +} - {~}.

Definition 6 (the set of relevant weights) Given an object o c V, and a path p that starts from o, we define the set of relevant weights to be the ordered set W S j , p = {xix E WSo,p[j], for some j A x r WSo,p[i]Vi " i < j A ~x r WSo,p[i]Vi " i < j}.

For instance, in the example of figure 1, we obtain the following set: W SClyde,Clyde_+ noyai Elephant_+ Elephant_+Gray -- { (--, Gray)}

Definition 7 (support of a fact due to an inheritance path) Given an object o E V, a path p that starts from o, an arbitrary set o f f acts S = {(+,k)ik E V} and an object c E C, we define the predicate edg_supo,p(c, S), with the meaning that isl(o, c), as follows: edg_supo,p(C, S) is true iff (+, m) e S (therefore isl(o, m)), where m is the direct predecessor of c along the path p.

For instance, in the example of figure 2, edg-supo,o~M-,c-~D-,E(C, {(+,O), (+, M)}) is true but not the edg_supo,o--~,-,c~r)--~E(E, {(+, 0), (+, M), (+, C)}). Definition 8 (support of a fact due to a positive weight) Given an object o c V, a path p that starts from o, a set of facts S = { ( + , k ) i k C V} and an object c C C, we define the predicate weight_supo,p(c, S), with the meaning that isl(o, c), as follows: weight_supo,p(c,S) is true, if there exists a weight (+,c) E WS~,p that is attached to an edge (k,l) and (+,k) C S but there does not exist a weight ( - , r ) E WS~,p such that it precedes (+, c) and r is both a successor of k and a predecessor of c.

As we will see in the next section, this definition precisely supports generic stability for WINs. For instance, w e i g h t - s u p o , o - ~ r . ~ R ~ T - , c ( C , {(+, O), (+, K), (+,L)}) is true in the example of figure 2, but it would not be true if a weight ( - , R) was attached to the edge (O, K).

47 Definition 9 (support of a fact) Given an object o E V, a path p that starts from o, a

set of facts S = {(+, k)lk e V} and an object c e C, we define the predicate supo,p(C, S), with the meaning that isl(o, c), as follows:

-2) ..... I C" ~176

(-,T .~

........

(+,C)~~~

Figure 2. A WIN

Definition 10 (the set of characteristics)

Given an object o C V, and a path p that starts from o, we define, inductively, the set of characteristics of o to be the maximal ordered set CS as follows:

{+,o}

CSo,, =

{x = (~, c ) l ( x 9

ifi=O

ws:,, A ~ =

-)v

p ))) ((x e LSo,p A-~x r WS~,p) A SUpo,p(c, CSo,~-1

otherwise

For instance, in the example of figure 1, we obtain the following sets: 0 C S Clyde,Clyde-+RoyalElephant--+Elephant-+Vray -- { ('+', Clyde)} 1 C S Clyde,Clyde-+ RoyaI Elephant--+ Elephant-+Gray - { (+, Clyde), (+, RoyalElephant), (-, Gray) )

2 C S Clyde,Clyde-+ Royal Elephant-+ Elephant -+Gray - { (+, Clyde), (+, RoyaIElephant), (+, Elephant), (-, Gray) ) 3 C S Clyde,Clyde-+ RoyalElephant-+ Elephant-+Gray - {(+, Clyde), (+, RoyalElephant), (+, Elephant), ( - , Gray)}

48

3. H A N D L I N G

REDUNDANCY

Since our aim is to simulate commonsense reasoning as closely as possible, it is important for a nonmonotonic reasoner to handle redundant information in a consistent manner. Of course, as long as a reasoner's knowledge about the world remains unchanged in time, this kind of information could be recognized and, subsequently, removed during a preprocessing phase of that reasoner's life cycle. However, if the reasoner's knowledge is continually subjected to revisions, as it is often the case, it is always possible that redundant, or even contradictory, information is introduced as a side effect of a revision process. Under our formalism, a weight is used to represent redundancy, in a way similar to the way a weight is used to represent an exception. This has as an important effect, the preservation of the uniformity of the inheritance network links as well as the preservation of the important "non-shortcut" property.

~ ~

w

o

/-

MaleRoyalElephant RoyalElephant

Clyde

(+,Elephant)

Clyde

~

~_

(-,Gray)

MaleRoyalElephantRoyalElephant

Elephant

Gray

~.

~,~

Elephant

Gray

Figure 3. A WIN with redundancy

Redundant information is closely related to the stability of a reasoner. Suppose that a reasoner, whose knowledge is represented by an inheritance network, supports the conclusion isl(X, ]I). We say that the reasoner is stable if, after inserting into its knowledge base the (redundant) information that isl(X, ]1"), it continues to support all the conclusions it supported before the insertion of this piece of information. Consider the example illustrated in figure 3. This inheritance network is supplemented with the "redundant atomic statement" isl(Clyde, Elephant). Since, the edge (Clyde, MaleRoyalElephant) has the weight (+, Elephant), we obtain the following sets, where o=Clyde and p=Clyde ~ MaleRoyalElephant ~ RoyalElephant --+ Elephant --+ Gray:

9 LSo,p = {(+, Clyde), (+, MaleRoyalElephant), (+, RoyalElephant), (+, Elephant), (+, Gray)}

9 WSo,p = WS~,~ = {(+, Elephant), (-, Gray)}

49 From these two sets, we form the set CS"

9 CS4o,p= {(+, Clyde), (+, MaleRoyalElephant), (+, RoyalElephant), (+, Elephant), (-, Gray)} Note that (+, Gray) is not included in this set, although it belongs to LSo,p and it holds that supo,p(Gray, CS4o,p). This is because (-, Gray) belongs to WSj, p. In shortest-path reasoners (see, for example, [8]), the addition of a redundant link alters the semantics of the inheritance network. In the previous example, a shortest-path reasoner supports the conclusion isl(Clyde, Gray), since the path Clyde -+ Elephant --+ Gray is shorter than the path Clyde --~ MaleRoyalElephant --+ RoyalElephant -~ Gray. If the redundant link were absent, we would conclude that nisl(Clyde, Gray), since the path Clyde -+ MaleRoyalElephant --~ RoyalElephant -/+ Gray would be shorter than the path Clyde -+ MaleRoyalElephant ~ RoyalElephant --+ Elephant -+ Gray). In [12] this kind of stability, called atomic stability, is viewed as an acceptability criterion for an inheritance reasoner. However the other kind of stability, called generic stability, is not viewed as such a criterion. Consider the inheritance network illustrated in figure 4. It shows a WIN supplemented with a "redundant generic statement" ([12]).

.

Clyde

.

.

.

.

-

RoyalElephant x-

Clyde

.

(-,Gray)

RoyalElephant

.

Elephant

Gray

InVisiblelnGrayBackground

x (+,InVisiblelnGrayl~kground) Elephant

Gray

InVisiblelnGrayBackground

Figure 4. A redundant generic statement

In the absence of the redundant information isl(Elephant, In VisibleInGrayBackground) it is (correctly) concluded that nisl(RoyaIElephant, In VisibleInGrayBackground). However, after inserting this redundant information, both shortest-path reasoners and skeptical reasoners conclude that isl(RoyalElephant, In VisibleInGrayBackground). This is, clearly, a false conclusion since royal elephants should always be visible in a gray background because they are not gray, regardless of whether it is directly stated or not that elephants are not visible in a gray background. Similarly, Clyde is not gray regardless of whether it is directly stated or not that Clyde is an elephant. Therefore, in our opinion, both atomic and generic stability should be viewed as acceptability criteria for an inheritance reasoner. WINs satisfy both of these criteria. In the previous example, the fact isl(RoyalElephant, In VisibleInGrayBackground) is not supported because, although isl(RoyalElephant, In VisibleInGrayBackground) belongs to LSo,p (where o=Clyde

50 and p--Clyde -+ RoyalElephant --+ Elephant ~ Gray ~ InVisibleInGrayBackground), SUpo,p(InVisibleInGrayBackground, CS4,p) is not true because neither edge_supo,p(In VisibleInGrayBackground, CSno,p) nor weight_supo,p(InVisibleInGrayBackground, CSno,p) is true. This latter fact is due to the weight (-, Gray) that takes precedence over the weight (+, InVisibleInGrayBackground) according to definition 4, and Gray is both a predecessor of In VisibleInGrayBackground and a successor of Elephant. Of course, one could argue that in the example shown in figure 5 generic stability prevents us from concluding that Moby is an air-breather. We believe that nisl(Moby, AirBreather) is a correct conclusion that is consistent with the knowledge that is represented by the inheritance network shown in this figure, since this network does not represent all of the information a man usually has about whales. The exception that whales, although not land-dwellers, are air-breathers, is not explicitly represented regardless of whether the fact that mammals are air-breathers is directly or indirectly represented.

dh, W

Moby lb,

Moby

Whale

Mammal

Land-Dweller

x_(-,Land-Dweller) x_(+,Air-Breather) ~ Whale

Mammal

Air-Breather ~,~

Land-Dweller

Air-Breather

Figure 5. Another redundant generic statement

The way in which redundancy is handled by WINs, defines the way in which conflicts due to exceptions are resolved. WINs do not support a mechanism analogous to the classical form of on-path preemption as this form is described elsewhere (see [12,18]). The semantics of WINs do not support off-path preemption either. Instead, they support a mechanism analogous to a restricted form of the classical on-path preemption, where the conclusion of a path is preempted only due to an edge (a weight in our formalism) of opposite polarity (sign) which emanates from (is attached to) a node (an edge) belonging to that path. Moreover, according to the classical form of on-path preemption, the conclusion of a path is also preempted due to an edge of opposite polarity which emanates from a node belonging to a path that starts from and ends at adjacent nodes of the path preempted. This second form of on-path preemption as well as off-path preemption are also used to support atomic stability for skeptical reasoners. However, they need not be incorporated in WINs since they support atomic and generic stability in a different way.

51 4. H A N D L I N G

MULTIPLE INHERITANCE

Nonmonotonic multiple inheritance frequently introduces multiple extensions of a theo.ry. We quote from [6]" " . . . a n extension is a set of beliefs which are in some sense "justified" or "reasonable" in light of what is known about a world". In the context of WINs, we associate an extension with a specific object of the represented world or system of beliefs (theory). Therefore, we restrict an extension to a set of beliefs about a world concerning (or obtained through knowing the existence of) that object. Formally, given an object o E V we define an extension E S for this object as follows: ESo,p, = {(+,o)U(s,n)l(s,n) e CSo,p,}, where CSo,p~ is defined for some path Pi that starts from o. Thus VES w.r.t an object o it is E S C_ Th((+,o)), where Th((+,o)) denotes all the properties of o that can be derived. The proposed inference consider all the facts that belong to any possible extension E S w.r.t, an object.

Pacifist (-,Pacifi~)~ Quaker

Republican ~

Nixon

Figure 6. Two conflicting extensions

In WINs, the beliefs obtained by passing through a path are included (either a subset or all of them) in an extension. Therefore, all cases of multiple inheritance are treated as cases of ambiguity, as it is also the case in [21] and [5], and there is no effort to resolve the possible conflicts as it is the case in [19]. The way in which multiple extensions are treated, depends on whether we adopt a credulous or a skeptical view. Consider the WIN shown in figure 6. Since there are two paths that pass through "Nixon", we obtain:

1.

,, LSlo,p, = {(+, Nixon), (+, Republican), (+, Pacifist)} 9 WS1 o,p~ = WSI~,p~ = { (-, Pacifist) }

,, CS12

= {(+,Nixon)(+

Republican)(-Pacifist)}

52 9 LS2o,v2 = {(4, Nixon), (+, Quaker), (4, Pacifist)} 9 WS2o,v ~ = WS2~,v2 = 0 2 9 CS2o,v~ = {(+, Nixon), (+, Quaker), (4, Pacifist)}

From the sets CS12

O,pl

and CS22op~ we obtain the following extensions, w.r.t Nixon: ,

'

1. ESlo,p, = {(+, Nixon), (+, Republican), (+, Quaker), (-, Pacifist)} 2. ES2o,v.~ = {(+, Nixon), (+, Republican), (+, Quaker), (+, Pacifist)}

This is because the conflict between the (-, Pacifist) of CS12,p, and the (4, Pacifist) of 2 CS2o,p~ cannot be resolved in favor of one or the other. This conflict, prevents the two C S sets from being contained both in the same extension. There is another additional feature of the proposed formalism that becomes obvious in the example shown in figure 6: the way in which negative links are treated. A WIN, as defined above, is a unipolar system (see [22] for the basic terminology), in the sense that negative (exception) links are not allowed. Nevertheless, WINs are capable of expressing negative links of classical inheritance networks through negative weights of the form ( - , c). For example, the negative link between Republican and Pacifist is expressed with the positive link (Republican, Pacifist) and the weight (-,Pacifist) attached to it. 5. T W O I M P O R T A N T

PROBLEMS

The sequential complexity of the goal-directed inheritance reasoning problem was examined in [18] when attacked by reasoners based on classical inheritance networks (see [17,21,12]). The problem was proven to be NP-hard for on-path/off-path credulous downward reasoners as well as for on-path upward credulous reasoners. On the positive side, the problem is in P for skeptical reasoners. In this section, we formally define the recognition problem, in the form presented in [19,7], and the recognition problem, for reasoners based on WINs. Definition 11 ( g o a l - d i r e c t e d i n h e r i t a n c e r e a s o n i n g ) Given a Weighted Inheritance Network W = (G(V, E), ~), an object o E I U C and a property/class p E C, decide whether the W I N supports the conclusion isl(o,p), nisl(o, p), neither or both. Definition 12 ( o b j e c t r e c o g n i t i o n p r o b l e m ) Given a Weighted Inheritance Network W = (G(V, E), jc), and a set of properties~classes sp, identify those objects o E I where W supports the conclusion isl(o, p)Vp C sp.

The central idea of the algorithms is to locate the most specific nodes which contain weights relevant to the properties under consideration (the nodes representing these properties can be among these most specific nodes). Intuitively, these nodes are the ones which decide inheritance according to the inferential distance measure, since they represent the most specific information. The fact that we can locate these nodes efficiently in a WIN, is the principal reason behind the good behavior of our algorithms. In the sections to follow, we consider parallel and hybrid algorithms that solve the problems defined above. The hybrid algorithms support both atomic and generic stability as

53

described previously. However, the current version of the parallel algorithms, supports neither atomic nor generic stability. Nevertheless, the parallel algorithms have the property that redundant links do not change previously derived conclusions, as it happens in skeptical or shortest-path reasoners, but additional extensions are derived instead. 5.1. Some definitions and n o t a t i o n Definition 13 An edge e = (i, j) E E is a p-edge if (•

E ~ ( e ) or e = (i,p), where p

is the property under consideration.

The edge (RoyalElephant, Elephant), in the example illustrated in Figure 1, is a p-edge as far as the property G r a y is concerned, since the weight (-, Gray) is attached to it. Definition 14 A p-path is a path containing a p-edge. In the previous example, the path Clyde --+ RoyalElephant ~ Elephant --+ Gray is a p-path because of the p-edge (RoyalElephant, Elephant). Definition 15 We say that an edge e = (i, j) E E or a node v C C is edge-reached by a node o E I U C if there exists a path (regardless of the constraints imposed by the weights of its edges) from o to i in W . These edges and nodes are called active.

Definition 16 We say that an edge e = (i, j) E E is weight-reached by a node o E I U C

iS: i) there exists a path q from o to i in W and ii) (fie' E q such that ( - , w) e ~'(e') and w is the label of an ancestor of i in q) or (~e' E q such that ( - , w) C ~'(e') and w is the label of an ancestor of i in q but 3e" C q such that e" precedes e' and (+, v) e 9V(e") and w precedes v and v precedes i).

Consider the node Gray in figure 1. This node is edge-reached by the node Clyde but it is not weight-reached by this node, since the weight (-, Gray) is attached to the edge (RoyalElephant, Elephant). 6. T H E P A R A L L E L

ALGORITHMS

We now present parallel algorithms for the goal-directed inheritance problem and the recognition problem. The algorithms are designed for the CRCW PRAM model of parallel computation (see [14]). Briefly, a PRAM is a parallel computer that consists of a number of processing elements that share a pool of memory, in addition to having a local memory area. The processing elements operate in lockstep, each executing the same instruction on different data, under global control. In the CRCW flavor of PRAM, any set of processing elements may read from or write into a memory location simultaneously. Bear in mind that the algorithms presented in this section are rather impractical in terms of processor requirements. However, they serve the purpose of illustrating the support that WINs offer for the development of efficient parallel algorithms. The departure point of our algorithm concerning the goal-directed inheritance problem, is to find the most specific p-edges w.r.t, the object o. The algorithm is based on digraph reachability, which is implemented by computing the transitive closure of the digraph

54 induced by the WIN. The reachability process is reduced to a breadth-first search (BFS) numbering of a directed acyclic graph. In [9], a parallel algorithm is given that solves this problem in O(log 2 n) time using O(M(n)) processors. After computing a single-source BFS numbering of the nodes, we can decide whether a node is reachable from the source by examining its distance from the source. The directed acyclic graph induced by the WIN is represented by an n x n adjacency matrix A, where n = II U C I is the number of labels of the WIN, and it is stored in the shared memory, before the beginning of the algorithm. If A[i,j] = 1, then there is a directed edge from node i to node j, whereas if A[i, j] = 0 then there is no such edge. It should be noted that whenever we use a variable with name Ay, we assume that we use a new copy of A, related to y. We have not included the copying procedure in our algorithm, since it can be easily done and its inclusion would only complicate the presentation. As a notational convenience, the statement for each ... is not followed by the processor indices of the processors it activates but, rather, by a symbol that represents the proper set of these processors. The algorithm is shown in figure 7. Before explaining its operation, we will list and explain some auxiliary procedures that are used by the algorithm.

9 Find_Reachable_From(o):

First, the algorithm identifies the subgraph of G that contains nodes reachable by the object under consideration. It uses the copy of the adjacency matrix that is stored in the shared memory.

9 Is_Reachable(x,y,z): This procedure returns true, if y is reachable by x, using the adjacency matrix specified by the parameter z.

9 Is_Active(x): It returns true, if x is an active edge or node (see definition 15). 9 ZeroOutColm(a,s): It write zeroes into the columns of the adjacency matrix a that correspond to nodes in the set s, i.e., it disregards all the edges of the WIN that are directed towards nodes in s.

9 Subset_Of(S): This operation computes all the subsets of the set S. It is used in the algorithm to activate appropriate sets of processors. First, the algorithm identifies the active nodes and edges (line 3). This is effected by a single-source multiple-sink reachability process taken from [9]. From that stage on, the predicate Is_Active(o) that is subsequently used, is well-defined and computable in constant time. In lines 5 through 7, all the plus weights, except those of the form (+, p), are converted to actual edges. Then the algorithm locates all the p-edges (lines 8 through 10). Consider, for example, the inheritance network shown in figure 8. In this particular example, the p-edges are denoted by labels that begin with "a". The next step is the elimination of those p-edges that are reachable by the node o only through other non p-edges. This is achieved in lines 11 through 17 by performing a single-source single-sink reachability process from the node representing the object o that is under consideration to each encountered p-edge. However, before this step is executed, the algorithm excludes from the corresponding copy of the adjacency matrix of each p-edge all the other p-edges (lines 12 and 14). In the previous example, only the edges whose labels begin with "ab" remain.

55 i p r o c e d u r e goal_dir ( W : W I N , o : i n t e g e r , p : i n t e g e r ) 2 begin

3 4

Find._Reachable_From(o) (* ldenti .fy active nodes *) (POS,NEG)+-(0,0)

5

f o r e a c h e--(i,j) E E:|s_Active(e) in p a r a l l e l do

6 7

for each (+,a)E ~'(e):aE V-{p} in p a r a l l e l do A[i,a]=l (* Convert positive weights to edges *) f o r e a c h e=(i,j)EE:ls_Active(e) in p a r a l l e l do if (-t-,p)E .T'(e) o r j = p type[e]='a' (* e is a p-edge *) for each e=(i,j)EE:type[e]='a' in p a r a l l e l do for each g=(k,1):gs~e a n d type[g]='a' in parallel do begin Ae[k,1]=0 if ls_Reachable(o,i,Ae) type[e]='ab' end for each e=(i,j) E E:ls..Active(e) in p a r a l l e l do (* Phase 1 begins *) i f 3 (-,a) E .~'(e):ls_Active(a) A'[i,j]=0 for each h=(k,l)EE:type[h]--'ab'in parallel do if Is_Reachable(o,k,A') begin type[h]='abc'; g o t o l i n e 52 end for each e--(i,j) E E:ls_Active(e) in p a r a l l e l do (* Phase 2 begins *) if 3 (-,a) E ~'(e):ls_Active(a) Zero_Out_Colm(A', { a}) for each h=(k,1)EE:type[h]='ab' in parallel do if ls_Reachable(o,k,A ' ) begin type[h]='abc'; g o t o l i n e 52 end for each e=(i,j) E E:ls_Active(e) in p a r a l l e l do (* Phase 3 begins *) if 3 (-,a) E _~'(e):ls_Active(a) S+-SUa for each X E Subsets_Of(S) in p a r a l l e l do begin for each e=(i,j) E E:ls_Active(e) in parallel do if 3 (-,a) E ~(e):a E X Ax[i,j]--0 (* Remove edge e *) Zero_Out_C olm ( A x , {S-X }) for each h=(k,1)EE:type[h]='ab' in p a r a l l e l do if ls_Reachable(o,k,A x ) begin type[h]='abc'; g o t o l i n e 52 end end for each e=(i,j) E E:type[e]='abc' in parallel do if (-,p)E Y(e) NEG+-I

8

9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 else 56 POS+-I 57 c a s e (POS,NEG) is: 58 (1,0): say 'isl(o,p)' 59 (0,1): say 'nisl(o,p)' 60 (1,1): say 'isl(o,p) and nisl(o,p)' 61 (0,0): say 'nisl(o,p)' (* Under the "closed world" assumption *) 62 e s a c 63 e n d

Figure 7. The parallel algorithm for the goal-directed inheritance reasoning problem

56 The next stage is the elimination of those p-edges that are not weight-reached by o (lines 18 through 51). This elimination is achieved in three phases, each of which is executed in parallel for each p-edge with label "ab". In our example, only the nodes whose label is "abE" remain. In each phase, a different possible kind of path that may connect the nodes o and p is sought. The first possible kind of path (considered in lines 18 to 26), includes no edges with negative weights, that refer to active nodes. In our example, and considering the p-edge emanating from K, the path O --+ E --+ F --+ A --+ C ~ K is exactly such a path. The weight ( - , X) that is in the weight-list of edge (C, K) does not refer to an active node, and thus, it has no effect on the progress of the phase. The existence of such paths is examined by performing a single-source single-sink reachability process from o to the tail of each edge whose label is "ab" (line 22), after eliminating the edges whose weight-lists contain negative weights that refer to active nodes (lines 18 through 20).

~ ~176176176176176

", ",

-,P)a oOo.~176176176

( ,P)ab

A,

/ i

1~.

\

.~176 o.~176176176176

S

(,B)

(-,L)

Figure 8. A WIN

57 The second kind of paths are those that include edges with negative weights but do not include the nodes mentioned in these negative weights. The existence of such paths is checked in lines 27 through 35 by performing a single-source single-sink reachability process from o to the tail of each edge whose label is "ab", after excluding the nodes mentioned in negative weights (lines 27 and 29). In the WIN of figure 8 we can see that the nodes that have to be excluded are A, B, C and L. The path O -+ E -+ G --+ H -~ I ~ D --+ J -+ K is a path of the second kind. Of course, if at least one such path is found, the algorithm does not proceed to the last phase for this particular p-edge. Consider now what would happen if the edges (E,F) and (I,D) did not exist. That is, if there were no paths of the two kinds of paths mentioned above. The third kind of paths are those that include at least one node that is mentioned in at least one negative weight of some active edge that is not included in the path. The existence of such paths is examined in lines 36 to 51 where a single-sink reachability process from o to the tail of each edge whose label is "ab" is performed in parallel for each possible subset of all the nodes mentioned in the negative weights (lines 36 through 38). In the WIN of figure 8 all the subsets of the nodes A, B, C and L are examined against inclusion in the same path. During the examination of a subset, we exclude the following: 9 All the edges with attached negative weights concerning the nodes of the subset. 9 The rest of the mentioned (in weights) nodes that are not included in the subset. In the WIN of figure 8 there is only one subset that exists in a path. This subset consists only of A and the path is O --+ E -~ G ~ H -+ A --+ D -+ J -~ K. Before the examination of this combination, both the edge (E, M) and the nodes B, C and L are excluded. Obviously, if o is reachable from p in any one of the three phases, it is considered to be reachable from p. Finally, in lines 52 through 62, the most-specific p-edges that are weight-reached by the object o provide the answer. As for the complexity of the algorithm, it is not difficult to see that it is dominated by the complexity of the third phase. This phase takes O(log 2 n) time using O(2~eM(n)) processors, since there are e p-edges in the worst case, and for each one of them there are 2~ possible subsets that are examined, performing for each of them a single-source, singlesink reachability process. We observe that there is an exponential term in the processor complexity figure, that depends on the value of k, the number of nodes mentioned by negative weights. This k can be in the order of n. However, in practice, the number of exceptions that are contained in knowledge bases is expected to be rather low. If it happens that k is in the order of log n, then the algorithm is an NC algorithm, i.e. it is technically an efficient parallel algorithm (see [14]). Of course, the above complexity is the worst case complexity. In practice, a lot of heuristics can be applied in order to reduce the complexity of the algorithm. For instance, the number of different combinations can be considerably decreased if we take under consideration that some nodes can not be in the same combination because they are not reachable by each other. The algorithm concerning the recognition problem proceeds in parallel for each property/class p C sp. For each property, it takes time O(log 2 n) using o(2kM(n)) processors.

58 Therefore, the overall complexity of the algorithm is O(log 2 n) using O(2*pM(n)) processors. The steps, for each property/class, are implemented in a way similar to the one described above for the goal-directed inheritance reasoning problem. The marking of the objects which can be weight-reached from the node representing the property/class under consideration is achieved through a three phase single-source multiple-sink reachability process starting from p. The three phases are exactly the same as those described in the algorithm for the goal-directed inheritance reasoning problem. In the end, all objects reachable from the node p in any of the three phases are considered to have the property p. In the processor complexity figure shown above for the recognition problem, we observe that the factor e is absent, although this is a more difficult problem than the goal-directed inheritance reasoning problem. This is because we only consider whether an object has a property or not and we do not use the full definition of the goal-directed inheritance reasoning problem, according to which we also consider if it both has and does not have a property, or there is no conclusion at all. Similarly, if we restricted in the same way the goal-directed inheritance reasoning problem, the processor complexity of the corresponding algorithm would be O(2kM(n)). 7. W I N B R I D :

A HYBRID

REASONER

A hybrid algorithm is a sequential one that includes some steps that are executed in parallel. The hybrid algorithms for both problems defined above, are based on a reachability process that locates the most specific p-edges w.r.t, an object o. We implement this process using a scheme based on depth first search (or DFS for short). In the hybrid algorithm concerning the goal-directed inheritance reasoning problem, the reachability process is guided by the encountered weights and starts from the node representing the object o under consideration. The hybrid algorithm concerning the recognition problem is actually a modified version of the algorithm for the goal-directed inheritance reasoning problem that is repeated in parallel, in the worst case, for every object o E I. It proceeds in parallel for each property/class p C sp. In figure 9, this reachability process for the goal-directed' inheritance reasoning problem is shown (procedure answer). It consists, mainly, of the procedure dfs shown in the same figure, that accepts as inputs a WIN W = (G(V, E), $'), the previously visited node v and the property/class p. In order to implement the main reachability process, we modify the DFS algorithm to propagate the weight-list of the previously visited edge to the weight list of each edge it visits, whenever it is needed (steps 31 through 39). Meanwhile, since it may be the case that an already visited edge has to be visited again via another path, if the algorithm reaches a node and there are also other edges directed towards this node, then the algorithm backtracks temporarily. It will only visit the edges emanating from this node (step 12) after all of them have been visited (steps 54 and 55). Therefore, an edge may hold more than one weight-list. This is because it accepts, during the upward propagation of the weight-lists, the weight-lists of eve .ry edge incident on its head node (step 15). These additional weight-lists, are constructed when the edge is visited (step 30). Before being propagated upwards, the weight-lists of the currently visited edge are examined against the following cases:

59 1 p r o c e d u r e answer ( W : W I N , o : i n t e g e r , p : i n t e g e r ) 2 begin 3 dfs(W,o,p) 4 case ( P O S , N E G ) is: 5 ( N O N E M P T Y , E M P T Y ) : s a y 'iM(o,p)' 6 ( E M P T Y , N O N E M P T Y ) : s a y niM(o,p) 7 ( N O N E M P T Y , N O N E M P T Y ) : say 'i.r a n d ~.i.~l(o,p)' 8 ( E M P T Y , E M P T Y ) : s a y 'ni.~l(o,p)' (* U n d e r t h e "closed world" a s s u m p t i o n "1 9 end 10 p r o c e d u r e dfs ( W : W I N , v : i n t e g e r , p : i n t e g e r / 11 b e g i n 12 f o r e a c h edge ( v , w ) : v i s i t e d [ ( v , w ) ] = f a l s e d o 13 begin 14 visited[(v,w)]=true 15 f o r e a c h wght_llst of (a,v) E E 16 begin 17 ifv r 18 st at e = w g h t _ l ist ((a,v)) [0] 19 else 20 state=true 21 if s t a t e = t r u e (* T h a t is, t h e s t a t e is active "1 22 if i s . p _ e d g e ( ( v , w ) ) 23 begin 24 if ( + , p ) E w g h t _ l i s t ( ( v , w ) ) 25 POS=(v,w) 26 if ( - , p ) E w g h t _ l i s t ( ( v , w ) ) 27 NEG=(v,w) 28 continue 29 end 30 m ake_a_w ght_l i st ((v,w)) 31 if s t a t e = t r u e a n d v ~ o 32 begin 33 for each i 34 if wght_list ((a,v)) [ i ] ! = E M P T Y 35 w g h t . l i s t ( ( v , w ) ) [ i ] 4- wght_list((a,v))[i] 36 end 37 else 38 if state=false a n d v ~ o 39 w g h t _ l i s t ( ( v , w ) ) 4-- w g h t _ l i s t ( ( a , o ) ) 40 if 3 i: w g h t _ i i s t ( ( v , w ) ) [ i ] = ( - , w ) 41 42

43 44 z15 46 47 48 49

50 51 52 53 54 55 56

begin

wght_list ( ( v ,w) [0] = f a l s e faise_because_of=w 1ast.Jxeg_wei g h t = (-,w) end if 3 i: w g h t _ l i s t ( ( v , w ) ) [ i ] = ( + , w ) a n d s t a t e = f a l s e a n d ((+,w)w)) begin wgiat_list ( ( v ,w) [0] = t r u e false_because_of=NULL 1ast_n e g_wei ght = N U L L end end

i f V (u,w) E E v i s i t e d ( ( u , w ) ) = t r u e dfs(W,w,p) end

57 e n d

Figure 9. The reachability process for the goal-directed inheritance reasoning problem

50 1. If the currently visited edge is a p-edge, then if it has a positive sign we set a fixed memory location POS (steps 24 and 25), whereas if it has a negative sign we set a fixed memory location NEG (steps 26 and 27). Then we backtrack (step 28), since DFS ensures that we visit a most specific p-edge w.r.t, to o. 2. In case a ( - , j ) weight exists (clearly j r p) in the weight-list of the currently visited edge e = (i, j) C E, the algorithm enters an "inactive state" since, according to the semantics of WINs, the weight-reachability of the successor nodes of j must be precluded (steps 40 through 45). 3. If the algorithm is in the "inactive state" and the updated weight-list of the currently visited edge e = (i, j) E E contains a ( + , j ) weight, then the algorithm switches to its normal active state except for some special cases where the generic stability should be satisfied (steps 46 through 52). While the algorithm is in the "inactive state", it does not examine the first case (step 21) and simply propagates weight-lists without updating the weight-lists it receives (steps 38 and 39). The validity and completeness of the sequential algorithm follow from the properties of DFS. The algorithm does not write into the locations POS and NEG, unless there exists a p-edge that is reachable by o through, at least, one path containing no other p-edge. The reachability process is constrained by the encountered weights due to switching back and forth to the "inactive state". Visiting the nodes in depth first fashion takes O(n + e) time, assuming that n denotes the number of nodes and e the number of edges of a WIN. During the examination of each edge (v,w) the algorithm processes all the items of a number of weight-lists. This is because it may be the case that it propagates the weight-lists of all the visited edges incident on v. The number of these weights-lists can, in the worst case, be equal to the number of different paths in the WIN under consideration. But, as can be shown by an average case analysis, this number is generally not large. The rest of the operations in the step consist in searching a weight-vector, usually very short, for a specific property. Moreover, we need O(lI I + m) processors in order for the DFS scheme to be executed in parallel for eve .ry object o E I where m is the number of memory-processors (see the next subsection). Therefore, using O(n) processors, in the worst case, the overall performance of the algorithm is determined by the n + e factor. Notice that during the reachability process we locate the p-edges concerning all the properties in sp simultaneously. In case the currently visited edge is a p-edge concerning the property/class Px, if it has a positive sign then we set a corresponding memory location POS[x] whereas if it has a negative sign we set a different memory location NEG[x]. Finally, we backtrack since DFS ensures that we visit a most specific p-edge concerning px w.r.t, to o. In this case, what actually happens is a "virtual backtracking" since the DFS algorithm really proceeds, but without checking for a p-edge concerning p~. Instead, it checks for p-edges concerning the other properties/classes of the sp set. The DFS algorithm will check again for a p-edge concerning Px when it actually backtracks from the current edge.

61 7.1. T h e a r c h i t e c t u r e of W I N B R I D In order to evaluate the hybrid algorithms described above, we have implemented a reasoner, called WINBRID, that uses the algorithms in order to handle the two problems that were previously defined. WINBRID was implemented on the Parsytec GCel-3/512 parallel computer under the PARIX operating system. This is a MIMD machine that consists of 512 processors (T805 transputers) arranged in a 2-dimensional mesh. Considering the computational demands of our algorithm, this number of processors was rather very restrictive. This fact had considerable impact on some of our implementation decisions. The set of the allocated processors is partitioned into two sets: the ones that hold the adjacency matrix of the underlying graph (memory-processors) and the weight lists, and the ones that perform the reachability process for each object o E I (search-processors). As an implementation strategy, we store the weights attached to edges in the node from which the edges emanate, along with information that associates the edges with their weights. Since every node has an identification number, we exploit this number in order to represent negative or positive weights as positive or negative integers. At most n such numbers are included in a weight list, although in practical cases there are only a few of them. Each of the memory-processors, communicates directly with all the search-processors. Whenever a search-processor needs more information in order to proceed with DFS, it requests it from the memory-processor that possesses it. One of the allocated processors acts as the computation coordinator. During a preprocessing phase, it divides the information about the adjacency matrix and the weight lists among the memory-processors. This processor is also responsible for maintaining a balanced workload among the processors performing DFS. This is necessary whenever we have at our disposal fewer processors than individuals. Our tests were aimed at an average case analysis of the behavior of WINBRID. To this end, we used a variety of WINs with a wide range of numbers of nodes and weight-list sizes as well as graph depths. The WINs that were used in our tests were generated randomly, in accordance with the properties that define them, that is non-short cut property and acyclicity. The generation algorithm begins with the maximal acyclic directed graph with n nodes, where n is the number of objects in the WIN. In this graph, there is a directed edge from a node i to a node j only if i < j. Then the algorithm starts discarding edges randomly. A randomly selected edge is discarded only if the resulting graph is still connected and possesses the non-short cut property. This process is continued, until all the edges of the initial graph have been considered. The next step, is to add a weight list for each edge of the resulting graph. This is accomplished in a random manner too. This process is governed by an input parameter defining the maximum size of a weight-list. In figure 10, the speedup is shown as a function of the size of the WIN and the number of processors used. For these tests, we generated WINs of various sizes and ran the algorithm with 50 queries on the average on each of them, using different numbers of processors. We observe that the attained speed-up is very good for a mesh topology, where our experimental work was carried out. In figure 11, the performance of WINBRID is shown as a function of the maximum size of the weight-lists. We ran the algorithm for 50 queries, for a specific WIN with size n + e - 500 and for different maximum numbers of weights. We can conclude that the overall performance of WINBRID is not related to the number

52

45

|

|

|

|

I /

w

,

.i

n+e=9053 -.~.--:" n + e = 2 0 7 1 0 " : - + - - .~

40 I/ /

n,~,,~o2 . ~ z . r},i~/=42455~-~ ..... ...... b , e = ~ -~-.-

.13.S"

25

..............................

....... ..~ . . . . . . . . . . . . . . . . . . . .

15

.::::fir." 10

.-...~ ...... ::=: :..~=,,.,...,~:_-_.':.~ . . . . . . . . . . .

""

~ ........... o....,i ,,o~

o-~ ....

. ....................................

'

o...i

5

I

I number

I of processors

Figure 10. Speedup as a function of the size of the WIN and the number of processors

of weights attached to the edges but it, rather, depends on the number of processors used, as it can be seen in figure 10. In figure 12, the performance of WINBRID is shown for the goal directed inheritance reasoning problem using 64 processors. The time complexity of a query concerning the recognition problem, is reduced to the time complexity of a query concerning the goal directed inheritance reasoning problem, if there are sufficiently many processors. Therefore, figure 12 shows the performance attainable by the algorithm for the recognition problem if there are sufficiently many processors. 8. R E L A T E D

WORK

PARKA is a frame-based knowledge representation system that was first implemented on a SIMD computer and it is designed to provide extremely fast property inheritance and inference capabilities (see [7]). It uses a multiple inheritance structure, the Frame Network, to represent relationships among concepts. Each concept is represented by a suitable frame. PARKA uses a fast hybrid algorithm for the recognition problem with a complexity that depends on the depth of the Frame Network and it is O(n) in the worst case, using O(n 2) processors ([7]). However, there is a new and improved version of PARKA, running on a MIMD computer ([20]) with time complexity O(d, n/k), where d is the depth of the inheritance network, n is the number of frames and k the number of processors of the parallel machine. PARKA does not, currently, support exceptions of inheritance. Moreover, it is shown in [7] that PARKA does not completely implement Touretzky's "inferential distance ordering". Thus, there are some Frame Networks that

63 O.E!

0.25

0.2

0.15

0.I

0.05

0

20

40 Maximum ( W9

60

80

100

/ FAg9

Figure 11. Response time related to the number of weights

can not be handled by this system. In [12], an inheritance reasoner is presented that uses a hybrid algorithm for a PMPM machine. A PMPM (Parallel Marker Propagation Machine) is a SIMD machine which is composed of a number of processing elements each of which is assigned to a node or link of an inheritance network and is capable of executing a number of simple marker manipulation and propagation instructions. The communication structure of the machine is fixed and matches the topology of the inheritance network. The time complexity of this algorithm is O(n 2) in the worst case (although cases that give rise to worst case performance are very rare in practice), using O(n + e) processing elements. Although the reasoner described in that work implements a complete version of the "inferential distance ordering," it may fail to treat satisfactorily some inheritance structures that are supplemented with "redundant generic statements" ([12]). Moreover, the fixed topology of the machine makes it difficult to facilitate the dynamic changes that may be later imposed on the structure of the inheritance network. On the other hand, WINBRID supports multiple inheritance with exceptions, and implements a complete version of the "inferential distance ordering" measure supporting both atomic and generic stability, with a time complexity that it is determined by a factor in the order of n + e, using O(n) processors. Moreover WINPAR, a massively parallel reasoner that is currently under development, will provide massively parallel support for both the goal-directed inheritance reasoning and recognition problems, using the parallel algorithms described above.

54 =

I

0 0

50000

,oo'ooo

,so'ooo

I 200000

I 250000

300000

nodes+edges

Figure 12. Response time (goal-directed inheritance reasoning problem) related to the size of the WIN

9. F U T U R E W O R K The design of more efficient algorithms for both the goal-directed inheritance reasoning and recognition problems is our main research goal. More specifically, we intend to use an extension of some transitive compression technique in order to efficiently manage transitive relationships. There is already a sequential algorithm that uses such a compression scheme for the management of transitive relationships with exceptions (see [1]). This approach is directly compatible with the key idea behind Weighted Inheritance Networks. A parallel algorithm can be based on the parallel scheme presented in [10]. We are also currently adapting the parallel algorithms in order to implement WINPAR, a parallel reasoner capable of nonmonotonic, multiple inheritance reasoning. WINPAR is being implemented under the PARIX environment for software development, on Parsytec's GCel-3/512 parallel machine. Moreover, it is interesting to examine heuristics that may lead to improvements in the performance of the parallel algorithms. As far as the hybrid algorithm described in section 7 is concerned, there are some places within the procedure dfs where more parallelism could help, such as the parallel manipulation of the weight-lists. 10. A C K N O W L E D G M E N T S The authors wish to thank I. Andreopoulos, K. Stathelakos, and A. Tzelaidis for their help during the tests. The authors also wish to thank the High Performance Computer Laborato .ry for the use of their Parsytec GCel-3/512 machine.

55 REFERENCES

1. B. Boutsinas, "Efficient Management of Transitive Relationships with Exceptions", Technical Report, C.T.I., TR.95.11.40, 1995. 2. B. Boutsinas and G. Pavlides, Belief Revision in nonmonotonic multiple inheritance using Weighted Inheritance Networks, Proceedings, AI'93 Workshop on Belief Revision: Bridging the gap between theory and practice, Australia, Melbourne, 141-155, 1993. 3. G. Brewka, Incomplete Knowledge in Artificial Intelligence, Proceedings of 1st Workshop in Information Systems and Artificial Intelligence: Integration Aspects, Ulm, FRG, 11-29, 1990. 4. D. Coppersmith and S. Winograd, Matrix multiplication via arithmetic progressions, Proceedings of 28th Annual A CM Symposium on Theory of Computing, Marina Del Rey, 1-6, CA (1987). 5. D. Etherington and R. Reiter, On inheritance hierarchies with exceptions, Proceedings of the AAAI-83, Washington, 104-108, DC(1983). 6. D. Etherington, Formalizing Nonmonotonic Reasoning Systems, Artificial Intelligence, 31:41-85, 1987. 7. M. Evett, J. Hendler, and W. Andersen, Massively Parallel Support for Computationally Effective Recognition Queries, Proceedings of the AAAI-93, 297-302, 1993. 8. S. Fahlman, NETL: A System for Representing and Using Real-World Knowledge, MIT Press, Cambridge, MA, 1979. 9. H. Gazit and G.L. Miller, An Improved Parallel Algorithm that Computes the BFS Numbering of a Directed Graph, Information Processing Letters, 28:61-65, 1988. 10. E. Lee and J. Geller, Parallel Operations on Class Hierarchies with Double Strand Representations, Proceedings of the third Parallel Processing for Artificial Intelligence Workshop of IJCAI-95, Mondreal, Canada, 107-119, 1995. 11. J. Horty, R. Thomason, and D. Touretzky, A Skeptical Theory of Inheritance in Nonmonotonic Semantic Networks, Proceedings of the AAAI-87, Seattle, Washington, 358-363, 1987. 12. J. Horty, R. Thomason, and D. Touretzky, A Skeptical Theory of Inheritance in Nonmonotonic Semantic Networks, Artificial Intelligence, 42:311-318, 1990. 13. J. J~ Js An Introduction to Parallel Algorithms, Addison Wesley, Reading, Massachusetts, 1992. 14. R.M. Karp and V. Ramachandran, Parallel algorithms for shared-memory machines, in: J. van Leeuwen, ed., Handbook of Theoretical Computer Science (Elsevier, Amsterdam, 1990). 15. B. Kettler, J. Hendler, W. Andersen, and M. Evett, Massively Parallel Support for Case-based Planning, Proceedings of the 9th CAIA, Florida, 3-8, 1993. 16. T. Krishnaprasad, M. Kifer, and D. Warren, On the Declarative Semantics of Inheritance Networks, Proceedings of the IJCAI-89, Detroit, 1099-1103, MI(1989). 17. E. Sandewall, Nonmonotonic Inference Rules for Multiple Inheritance with Exceptions, Proceedings IEEE 7~, 1345-1353, 1986. 18. B. Selman and H.J. Levesque, The Tractability of Path-Based Inheritance, Proceedings of 11th IJCAI, 1989.

55 19. L. Shastri, Default Reasoning in Semantic Networks: A Formalization of Recognition and Inheritance, Artificial Intelligence, 39:283-355, 1989. 20. K. Stoffel, J. Hendler, and J. Saltz, PARKA on MIMD-supercomputers, Proceed-

ings of the third Parallel Processing for Artificial Intelligence Workshop of IJCAI-95, Mondreal, Canada, 132-142, 1995. 21. D. Touretzky, The Mathematics of Inheritance Systems, (Morgan Kaufmann, Los Altos, CA), 1986. 22. D. Touretzky, J. Horty, and R. Thomason, A Clash of Intuitions: The Current State of Nonmonotonic Multiple Inheritance Systems, Proceedings of the IJCAI-87, Milan, 476-482, Italy(1987). 23. W. Woods, What's in a Link: Foundations for semantic networks, in: R. Brachman and H. Levesque (eds.), Readings in Knowledge Representation, 217-241, 1985.

67

Basilis Boutsinas Basilis Boutsinas was born in Cefalonia, Greece, in 1963. He received his diploma in Computer Engineering and Informatics in 1991 from the University of Patras, Greece. He also conducted studies in Electronics Engineering at the Technical Education Institute of Piraeus, Greece and Pedagogics at the Pedagogical Academy of Lamia, Greece. He finished the preparation of his Ph.D. thesis at the University of Patras in 1996. His primary research interests include Knowledge Representation techniques, NonMonotonic Reasoning and Parallel Algorithms. He also works in the application of Knowledge Representation schemes in building educational systems. He is a member of AAAI and AACE. He can be reached at the University of Patras, Department of Computer Engineering and Informatics, 26500, Rio, Patras, Greece. Home Page: http://socrates.ceid.upatras.gr/-vutsinas

Yannis C. Stamatiou Yannis C. Stamatiou was born in Volos, Greece, in 1968. He graduated from the department of Computer Engineering and Informatics, University of Patras, in 1990. He is currently working towards a Ph.D. in computer science. His main research interests are in parallel and distributed algorithms for problems stemming from Artificial Intelligence, and especially Constraint Satisfaction Problems, and techniques for analyzing their expected performance. He is currently working on the problem of the satisfiability threshold for random formulae as well as on the extension of the problem for random Constraint Networks. He is also interested in Artificial Life, Fractals, Philosophy of Science and the roots of Computability and Logic. He is a voting member of ACM, IEEE, the Greek Computer Society, and the Greek Technical Chamber. He can be reached at the University of Patras, Department of Computer Engineering and Informatics, 26500, Rio, Patras, Greece. Home Page: http://fryni.ceid.upatras.gr/-stamatiu

Georgios Pavlides Georgios Pavlides is an associate professor in the Department of Computer Engineering and Informatics of the University of Patras, Greece. His research interests are in Software Engineering and Artificial Intelligence and their applications in Management Information Systems. He received his Ph.D. in Operational Research from the University of Sofia. He can be reached at the University of Patras, Department of Computer Engineering and Informatics, 26500, Rio, Patras, Greece.

Parallel Processing for Artificial Intelligence 3 J. Geller, H. Kitano and C.B. Suttner (Editors) 9 1997 Elsevier Science B.V. All rights reserved.

Parallel Operations

69

on Class Hierarchies with Double Strand

Representation Eunice (Yugyung) Lee* and James Gellert Department of Computer and Information Sciences New Jersey Institute of Technology Newark, NJ 07102 This paper continues a series of papers dealing with the problems of (1) fast verification of the existence of a transitive relation in an IS-A hierarchy, and (2) dynamic update of such a hierarchy. As in our previous work, a directed acyclic graph (DAG) of IS-A relationships is replaced by a set of nodes, annotated by number pairs, and stored on a massively parallel computer. In this paper a new mapping of this set of nodes onto the processors is described, called the Double Strand Representation (DSR). The DSR improves the processor usage compared to our previously used Grid Representation (GR). This paper shows IS-A verification and number pair propagation algorithms for the Double Strand Representation. Test runs on a CM-5 Connection Machine a are reported. 1. I N T R O D U C T I O N Most Knowledge Representation systems, as well as all object-oriented languages and databases, use an IS-A hierarchy as their backbone. An efficient hierarchy encoding technique would aid all these areas, especially for large hierarchies. The IS-A hierarchy has been especially important in the KL-ONE family of Knowledge Representation (KR) systems [1-9]. Object-oriented systems, based on SIMULA [13] and Smalltalk [14], always incorporate generalization hierarchies with inheritance behavior. Object-oriented methods have been applied to the design of programming languages, e.g., C + + [15], type systems [16], objectoriented extensions of existing languages, e.g., CLOS [17], and object-oriented database systems such as ORION [~0], o~ [~], and ONTOS [12]. Given the importance of the IS-A hierarchy, one would like to achieve the fastest possible processing for query and update operations in this hierarchy. If we assume that it is known that a Mammal is an Animal, a Dog is a Mammal, and a Collie is a Dog, then we want *This research was (partially) done under a cooperative agreement between the National Institute of Standards and Technology Advanced Technology Program (under the HIIT contract, number 70NANB5H1011) and the Healthcare Open Systems and Trials, Inc consortium. tThis work was conducted using the computational resources of the Northeast Parallel Architectures Center (NPAC) at Syracuse University, which is funded by and operates under contract to DARPA and the Air Force Systems Command, Rome Air Development Center (RADC), Griffiss Air Force Base, NY, under contract# F306002-88-C-0031. aThe Connection Machine is a trademark of Thinking Machines Corp.

70

Thins ( Animal ( Mammal(

M a m ~

(_) Reptile

Dog ( Collie ( )

Collie

k_) Cocker U Spaniel

Figure 1. IS-A Hierarchy

to quickly answer "yes" to the query whether a Collie is an Animal (Figure 1). We also want to be able to quickly update the hierarchy, e.g., when adding the facts that Cocker Spaniels are Dogs, Cats are Mammals, and Reptiles are Animals. In the past we have been especially interested in techniques where the response time for an IS-A que .ry does not depend on the length of the chain of IS-A links that must be traversed to answer the query. Besides these techniques, our tool of choice for achieving fast query and update operations is fine-grained parallelism. This raises the question of how to map the IS-A hierarchy onto the available space of processors. The most obvious intuitive choice is to assign eve .ry class of the hierarchy to a single processor. However, this intuitive choice does not carry over to the links between classes. If the whole hierarchy were known at the beginning of system design, one could opt for a strong form of isomorphism, where every IS-A link is implemented as a hardware link. However, our basic assumption is that Artificial Intelligence is not intelligence at all, if knowledge structures cannot be updated dynamically. Therefore, the isomorphism solution would require dynamic hardware changes as part of any update of the IS-A hierarchy, a solution that is currently still not practical. The idea of custom-made hardware is also not appealing to us. The solution that we have been using in a series of papers [23,25-30] has been to eliminate the need for the IS-A links as much as possible, while still maintaining all the knowledge that is contained in the IS-A hierarchy. In this way, we d o n o t have to worry about the mapping of the IS-A links onto the actual hardware. Like other researchers, we are representing the classes of an IS-A hierarchy by nodes in a graph. The IS-A links are represented by the directed edges of this graph. In our first paper on the subject [26] it was shown that for the special case of a tree-shaped IS-A hierarchy of nodes, the hierarchy could be replaced by a linear order of the same nodes, together with o n e number pair assigned to each node. The assignment of number pairs was based on an encoding due to Schubert [31]. Later on we extended the representation of IS-A hierarchies "without explicit IS-A

71

Figure 2. Three Step Mapping

links" to directed acyclic graphs (DAG). In this representation, several number pairs became necessary at some nodes. The assignment of these number pairs was based on an extension of [31] by Agrawal et al. [18]. While doing this, we were able to prove that the linear order used in [26] is not necessary at all. Rather, a set of nodes with an associated number pair(s) at each node could perfectly represent a DAG-shaped IS-A hierarchy without explicitly maintaining the IS-A links [23]. Unfortunately, the original fast algorithms [26] were possible due to the fact that one n u m b e r pair was assigned to one processor, and not due to the fact that one node was assigned to one processor. So, in order to maintain the speed of processing, at least for queries, it became necessary to change the mapping of nodes onto processors. In [23,28] we mapped each node onto one column in a two-dimensional grid of processors. Every number pair of each node was assigned to a different processor (row) in its column. A pleasant side effect of eliminating explicit links is that the time necessary to traverse them is also eliminated, giving, within certain limitations, constant time responses for transitive closure queries [26]. In other words, by using the Schubert/Agrawal representation, it takes as much time to verify that a Collie is an Animal as it takes to verify that a Collie is a Dog. By adding parallelism, updates can be performed in "almost" constant time. Experimental verification of this claim was provided in [26] for the case of trees. All necessary details of Agrawal et al.'s encoding, our node set representation, and the GR will be described later on in this paper. However, we summarize now that the main feature of our previous work is a three step mapping (Figure 2). In the first step, an IS-A

72 T "l~ng

Mineral

Animal

Domestic-

Siamese

Cheetah

Figure 3. A Class Hierarchy

hierarchy of classes of the real world is mapped onto an isomorphic DAG of nodes, with one class per node. In the second step, the hierarchy of nodes is mapped into a set of those nodes, so that eve~. node is annotated with one or more number pairs. In the third step, this node set and the associated number pairs are mapped onto the processor space of a fine-grained parallel computer. In brief, class hierarchy --4 directed acyclic graph --4 node set + number pairs ~ processor space. In our previous papers, the third step was performed by organizing processors as a twodimensional grid, with one node per column, and one number pair per row. Unfortunately, the GR causes a number of difficulties. We will describe the major problem now, while mentioning some other problems later in the paper. Because some nodes have only one number pair, while others have many number pairs, some columns might be virtually empty, while other columns might run out of processors, disrupting the functioning of our algorithms. Therefore, in this paper, we are showing a different representational approach. The general idea of a three step mapping class hierarchy --4 DA G --4 node set + number pairs --4 processor space is still maintained. Indeed, the first two steps of the mapping are not changed at all. However, the assignment of number pairs to processors is changed in a way that eliminates the main problem of the GR described above. The new representation is called Double Strand Representation (DSR) and forms the main subject of this paper. In addition, we will show parallel algorithms for fast IS-A queries in the DSR. We will also show parallel algorithms for an important operation of Agrawal et al.'s encoding [18], called propagation, in the DSR. Propagation is a necessa .ry part of eve~. update operation on a DAG with number pair annotation. Work related to ours in the symbolic paradigm has been published, e.g., [32,33,39]. The PARKA system, a symbolic approach to combining KR with massive parallelism, has been described there. It is a frame system for handling large amounts of knowledge. It is implemented on the Connection Machine, and its temporal behavior has been extensively

73

Class Name Thing

Animal

Plant Mineral

Mammal

Thing [ 1 10]

Number Pair

[1 [3 [2 [ 10 [5

10] 9] 2] 1O] 8]

Domestic-Animal

[9 9](8 8)

Wild-Animal

[4 4](7 7)

Feline

[6 8]

Siamese

[8 8]

Cheetah

[7 7]

k.~ [10 1 0 ~ Domcstic-~al

[9 9] 0

(8 8 ) ~ ,

8,si o o

,•

[2 2~-J

Mammal ",~ild-Animal 0[4 4] 4 [ ( 7 7) Feline ,

( )[5 8]

[7 7]

w

Figure 4. Node Set Representation for Class Hierarchy

tested. The newer version runs on an IBM SP2 [34]. Neural network approaches close in spirit to ours are, e.g., [40,42-46]. Shastri's work [40,41] combines massive parallelism implemented on a neural network simulator with a well defined, limited inference approach. According to Shastri [42], the distinction between the processes of a special-purpose reasoner and a general-purpose reasoner is akin to two human modes of reasoning, namely, reflexive reasoning and reflective reasoning. Sun [44], on the other hand, presents an intensional neural network approach of reasoning based on the semantic closeness of concepts. His work implements inheritance employing massive parallelism. In Section 2 we discuss the numeric encoding of class hierarchies and the node set representation. In Section 3 the Double Strand Representation is presented. In Section 4 constant time subclass verification in the Double Strand Representation is discussed. Section 5 presents massively parallel propagation algorithms with the Double Strand Representation. We compare the performances of both representations, giving experimental results on the Connection Machine in Section 6. Finally, we conclude this paper in Section 7.

2. N O D E SET R E P R E S E N T A T I O N A N D A G R A W A L / S C H U B E R T E N C O D ING Our work has been based on Agrawal et al.'s [18] extension of Schubert's class hierarchy reasoner [31] towards directed acyclic graphs. Agrawal et al.'s approach [18] makes it possible to verify that A is a subclass of B by comparing a number pair at A to one or more number pairs stored at B. This approach makes no use of the path from A upwards to B at all.

74

!

~g

, Path

Left Part

t Part

! 9

I I

! I

al I

l

I

I I

I I

i Domestic-Anita I

ild-Animal .

.

.

.

.

.

;

. . . . .

!

O ~ Siamese

.

i I

r~F~ine- -

-

"i

i Subtrcc

'I I

I

'i U~ Ch~,etah I

Figure 5. The Four Areas of Spanning Tree

The basic approach of [18] is: (1) Construct an optimal spanning tree of a given b A G such that at every node with multiple parents, we select the link to the parent with the maximum number of predecessors. Predecessors are nodes that are reachable from a node by an "up search." (2) Assign a pair of preorder and maximum number to every node. Preorder numbers are generated by a right-to-left preorder traversal of the spanning tree. The maximum number for every node is the maximum preorder number in the subtree rooted at that node. Tree pairs result from this step. (3) All the arcs that are not part of the optimal spanning tree are used to propagate number pairs upward. This is done so that every transitive IS-A relation in the bAG can be verified, but no redundant pairs are generated. Graph pairs result from this step. In Figure 4 we use the notation [Tr #] for tree pairs and the notation (Tr #) for graph pairs. In this representation a node A is a subclass of a node B iff the tree pair of A is included in or equal to any one of the pairs of B. For instance, a Cheetah is a Feline because [7 7] is a subinterval of [6 8]. A Cheetah is also a Wild Animal, because the tree pair of Cheetah [7 7] is propagated to Wild Animal as a graph pair (7 7). However, a Cheetah is not a Mineral because [6 8] and [10 10] are disjoint. In [23] our incremental massively parallel encoding of bAGs, called "node set representation," was introduced. We proved that the node set representation together with the number pairs is sufficient to represent a class hierarchy. We can operate with a set of nodes because all important update and retrieval operations require only three items at every node: (1) the key item, e.g., Mammal, (2) the number pairs, and (3) the area of the spanning tree where the node is located [23]. It is easy to see that the tree pair at each node N can be used to determine four areas of the graph (Figure 5). Every node N, except for the root, defines a path of spanning tree arcs that connect N to the root. This path divides the spanning tree into four (possibly empty) areas: (1) the path itself; (2) the left part of the path; (3) the right part of the path; (4) the subtree which is rooted at

75

[1 10] [2 2] 0

1

[3 9]

. . . . .

[8 81 lit9 91 ]ltlO ioJll""

2 ..... 7 Tree Pairs Strand

8

9 (Pq;J

|1 t9 911 (8 8) I Ira 41 (7 7) | Maxld-3 Maxld-2 MaxId- 1 Maxld ((I~ Graph Pairs Strand

Figure 6. Double Strand Representation for Figure 4

N. For instance, for Mammal in Figure 5, we can easily define area 1: {Thing, Animal, Mammal}, area 2: {Mineral, Domestic-Animal}, area 3: {Plant, Wild-Animal}, and area 4: {Feline, Siamese, Cheetah}. Many important steps of the update operations treat each of these four areas uniformly, with the same operations being applied to all the nodes in one area. Therefore, if we have area information, we do not need the class hierarchy any more. Details can be found in [23]. 3. D O U B L E S T R A N D

REPRESENTATION(DSR)

Now we will show how the node set is practically mapped onto the processor space of the CM-5. We are interested in a mapping that will represent tree pairs and graph pairs efficiently, so that it is possible to achieve a high degree of parallelism, memory efficiency, and optimal use of available processors. We call the result of our new mapping the "Double Strand Representation" (DSR). In this representation the given processors are divided into two areas: the tree pairs strand and the graph pairs strand (See Figure 6). In the tree pairs strand, every node is represented in a separate processor. The tree pair of a node may be assigned to the tree pairs strand in any order. In the graph pairs strand, pairs of processors are used to store a sequence of pairs, each consisting of a tree pair and a graph pair. Every processor is assigned an address, called its ID. Let source of propagation be a node which propagates its tree pair and let target of propagation be a node to which a number pair is propagated from the source of propagation. The tree pair U stored in a processor with an odd ID x is used to represent the target of propagation. The graph pair V in the processor with ID x + l is used to represent the source of propagation. Let Z be a processor pair (U, V) in the graph pairs strand. Let Y be the set of all Z. Every time a pair V is propagated to a node with tree pair U, we will represent (U, V) in the graph pairs strand. If 1K processors are available, the maximum ID (MaxId) will be 1022 in the DSR, and the first processor pair (U, V) will be stored in the two processors 1021 and 1022. In Figure 6 (representing the hierarchy of Figure 4) the tree pair of Wild-Animal [4 4] occurs as U in the graph pairs strand in processor MaxId - 1 (1021) and a propagated pair (7 7) which is the tree pair of Cheetah occurs in the even processor MaxId (1022) as a graph pair of Wild-Animal. Therefore, we can verify that Cheetah is a subclass of Wild-Animal. (Details will be shown later.)

76

Thing

Plant

Animal

W-Ani

Mammal

Feline Cheetah Siamese D-Ani

o 1[1 lO]I 1[2 2] I [tY'~ 1[4 41 I 115 811 I [6 811 1[7 711 ~ 1 I'

i

I ~

I!

i

11

I I (7 7)1 I

II

IF--G[

II

Mineral

i[9 911 1[1010]1

II

I F-

I (8 8)1 L___]

II

I1

I!

il

Figure 7. Grid Representation of Figure 4

We will now compare the Double Strand Representation to our Grid Representation used in previous research [27]. The Grid Representation is a distributed representation of a node set (Figure 7). Every node is represented as a column. The first row contains the tree pair of the node, while up to k graph pairs are maintained in the other rows. Nodes may be assigned to columns in the order that the system is informed about their existence. As the Double Strand Representation, this order is irrelevant. We have been using a grid of 128 columns and 8 rows. On our Connection Machine 1 2 8 , 8 = 1024 is the minimum number processors that may be used. The choice of 128 columns and 8 rows corresponds to a compromise between having a large node set and permitting a reasonably large number of pairs at each node. Note that these 1024 processors are "virtual," meaning that they are simulated on 32 real processors. While we could use larger sets of virtual processors, this would only slow down real-time results. Therefore, we needed to choose the minimum configuration. While using the grid structure, we encountered some problems. First of all, we have to allocate k processors for graph pairs for eve .ry tree pair. This causes a significant number of processors to be left empty. This can lead us to run out of processors when the number of graph pairs in one column exceeds k while in other columns processors are empty. Secondly, during update of the class hierarchy, we may have an unused processor between two used processors, called a "hole," because of our implementation of Agrawal et al.'s subsumption algorithm. Unfortunately, with the current algorithm for the grid structure, we have not found an efficient technique to reclaim such a hole. Agrawal et al.'s subsumption technique, as we have mentioned earlier, is an algorithm which eliminates subsumed pairs during propagation. For example, if a number pair (7ri pi)is subsumed by another pair [Trj #j] at the same node (column) due to propagation, i.e., 7r.i _< 7ri and #i <_ #j, then discard (7ri #i). It is due to this that some processors are left without pairs and become holes. All these problems have lead us to abandon the Grid Representation and turn to the Double Strand Representation. One question which arises now is: How do we efficiently organize processors into two strands? Suppose that we organize processors (by SW) in two rows; the first row is used to represent tree pairs and the second row for graph pairs. Since these two strands are growing at different rates of speed, we may encounter a case where all the processors in the tree pairs strand are used up while a lot of unused processors remain in the graph pairs strand. This is clearly not a good representation. It is necessary

1

77

Figure 8. Dynamic Storage Management of Double Strand Representation

to design a processor-efficient technique to avoid this problem. The main idea of our storage management is borrowed from the dynamic paradigm of languages such as Pascal that maintain a stack and a heap. In our representation, processors of the tree pairs strand are allocated starting at processor 0 and grow towards higher processor IDs. Processors of the graph pairs strand are allocated starting at the processor with the largest ID and grow towards lower processor IDs. There are two pointers, (I)~ and ~v, to show the borders of both areas. We define jc to be the size of "free space." Jr = ~)~ - ~)r

(1)

~" should be bigger than a certain threshold, say 10% of processor space. If this is not the case, we can take some corrective measures, to be described in [24]. 4. S U B C L A S S V E R I F I C A T I O N Now we will describe how to verify subclass relationships. Suppose that we want to verify whether B IS-A A. There are two cases: (1) A is a tree predecessor of B. (2) A is a graph predecessor of B. The first case can be easily verified by a subsumption test: the tree pair of A subsumes the tree pair of B. For the second case, we have to check whether A has a graph pair propagated from B or from a tree predecessor of B. We now introduce some CM-5 terminology. A parallel variable (pvar) is an array (perhaps multi-dimensional) where every component is maintained by its own processor and all values are usually changed in the same way and in parallel [47]. In the algorithm, variables marked with !! are parallel variables, and operations marked with !! or involving parallel variables are parallel operations. The parallel variable pre!! contains for eve .ry node a preorder number, and the expression max!! stands for a parallel variable that contains for eve .ry node a maximum number. The functions prenum(A), maxnum(A), and tree-pair(A) retrieve the preorder number, maximum number, and the tree pair, respectively, for the given node A. Additionally, the variable g-lb (Graph Strand Lower Bound) represents (I)~ and t-ub (Tree Strand Upper Bound) represents (I)~. The parallel function self-address!! returns IDs of all active processors and oddpll contains TRUE on a processor if the processor's

78 ID is an odd number. The algorithm ACTIVATE-PROCESSORS-WITH consists of two parts. The first part describes a set of processors to be activated. The second part, starting with DO, describes what operations should be performed on all active processors. We now show a function IS-A-VERIFY that performs subclass verification. As we mentioned above, if A is a tree predecessor of B (by IS-A-VERIFY-I) or A is a graph predecessor of B (by IS-A-VERIFY-2), then B IS-A A. ; B is-a A iff I S - A - V E R I F Y returns T. IS-A-VERIFY (B, A: Node)' BOOLEAN return(IS-A-VERIFY-I(B, A) OR IS-A-VERIFY-2(B, m))

IS-A-VERIFY-1 (B, A: Node): BOOLEAN ; If A is a tree predecessor of B, then the tree pair of A subsumes the tree ; pair of B. ACTIVATE-PROCESSORS=WITH prenum(B) >_!! prenum(A) AND!! maxnum(B)
Now we will show how to verify that B IS-A A when A is a graph predecessor of B. Remember that a pair of processors (U, V) in the graph pairs strand is used to represent a graph pair. The tree pair in the odd processor (U) is used to represent a node S and the graph pair in the even processor (V) is used to represent a node which propagates its tree pair to S. Therefore, we are looking for a pair of processors (U, V) such that the tree pair of A is contained in processor U and the graph pair of B or its tree predecessor is contained in processor V. In the following functions the expression mark!![x] "- y means that the pvar mark!! on the processor with the ID x is assigned the value y. IS-A-VERIFY-2 (B, A: Node)" BOOLEAN ; Activate every occurrence of the tree pair of A in the graph pairs strand. ; Set the parallel flag mark!! on the right neighbor processors of the active ; processors. ACTIVATE-PROCESSORS-WITH pre!! <__!! prenum(tree-pair(A)) AND!! max!! __!! maxnum(tree-pair(m))AND!! self-address!!() ___!! g-lb AND!! oddp!! (self-address!!()) DO BEGIN mark!![self-address!!() +!! 1]:= 1 END

79 ; Test whether any marked processor has the tree pair from B or ; from a tree predecessor of B, as a graph pair. If this is the case, return T. ACTIVATE-PROCESSORS-WITH pre!! <_!! prenum(tree-pair(B)) AND!! max!! :>!! maxnum(tree-pair(B))AND!! mark!![self-address!!()] =!! 1 DO BEGIN IF any processor is still active THEN return T END

In our example (Figure 6), Feline is a subclass of Animal because [6 8] is a subinterval of [3 9] (by IS-A-VERIFY-I). IS-A-VERIFY-2 will verify that Siamese is a subclass of Domestic-Animal because the tree pair [8 8] of Siamese will occur as (8 8) together with the tree pair [9 9] in the graph pairs strand. Feline is not a Plant because [6 8] is neither a subinterval of [2 2] nor is there a processor pair ([2 2], (6 8)) in the graph pairs strand. In summary, with the Double Strand Representation, it can be rapidly decided whether a subclass relation exists between two classes. 5. I N C R E M E N T A L

UPDATE

OF C L A S S H I E R A R C H Y

WITH DSR

In a directed acyclic relationship graph, there are two "obvious" incremental update operations: (a) inserting a graph component into another graph component, when both of them are initially disconnected components; (b) adding a new link between two nodes of the same graph component. We call (a) graph insertion and (b) link insertion, while insertion and update may refer to either one of them. We previously presented algorithms for graph insertion in [28] and link insertion in [23]. We need to show how the insertions can be performed with the DSR. The steps that have to be taken for an insertion consist of (1) the global number pair update [23] and (2) the propagation of number pairs. As shown before, in the Double Strand Representation, a class is represented by storing its tree pair in the tree pairs strand. Graph pairs are stored in pairs of processors in the graph pairs strand. Note that this makes the graph pairs strand a list of pairs of number pairs. In Section 2 we pointed out that all important update and retrieval operations require only three items of information at every node. The first two, the key item of the node and the number pairs at the node, need to be stored explicitly. The third item, the tree area, can be determined easily, as will be argued now. The global update operations (1) treat every number pair (whether it is a tree pair or a graph pair) in the two strands uniformly, with the same operations being applied to all the processors. The reason for that is as follows. The change of a graph pair has to mirror the change of the tree pair from which it was created by propagation. But how does the algorithm know which transformation to apply to a tree pair? It makes thisdecision based completely on the tree pair itself! Therefore, the same criteria can be applied to the graph pairs that are identical to a specific tree pair. It should be noted that we are not dealing with every possible transformation resulting from a global number pair update as this is the subject of a separate paper [23].

80 Now, we will discuss a parallel number pair propagation algorithm for (2) in detail. The propagation of number pairs is performed as follows. Suppose that a graph link is inserted from a node C to a node N. We have to propagate the tree pair of C to eve .ry predecessor Ni of N (including N itself). For every processor Ni to which a pair V is propagated we need to generate a new entry for the graph pairs strand. This new ent .ry consists of the tree pair of Ni and V: (Tree-Pair(N1), V), (Tree-Pair(N2), Y), ..., (Tree-Pair(Nk), V), where k is the number of predecessors. These newly generated pairs have to be assigned to 2 * k currently unused contiguous processors to the left of (I)~ (Figure 9(c, d)). If several pairs Vi need to be propagated, this process needs to be done serially. As an example (Figure 9(a, b)), due to inserting the arc from H to E, the tree pair V = [5 5] of H should be propagated to every predecessor of E, and E itself (namely, E, C, B, A). As m has the pair [1 8], we do not need to propagate [5 5] to A because [1 8] subsumes [5 5]. In our terminology, only E, C, and B are targets of propagation. For propagating [5 5], we need to find the appropriate IDs of processors to assign [5 5] to (in parallel). For this, we develop a parallel function to find proper processor IDs for each propagated pair (Figure 9(c)). First, we activate processors in the tree and graph pairs strands that correspond to predecessors Ni to which we want to propagate a specific graph pair V. Second, there is a parallel operation, enumerate!!, on the CM-5 that will assign numbers 0, 1, 2 ... to active processors. Third, we define a parallel function T to compute the processor ID where the processor with the number x (assigned by the enumerate function) should deposit its number pair.

T(x) = r

2(z + 1)

(2)

where x >_ 0 and x < ~ . T computes the odd position, and we generate the pair (T(x), T ( x ) + 1) for (Tree-Pair(N/), V). With these three steps, mapping each predecessor to its corresponding processor ID in the graph pairs strand can be completed. For instance, in Figure 9(a, b), when inserting the arc from H to E, we first activate every tree predecessor of E (C and E itself), but not A. Similarly we activate every graph predecessor of E (just B). Then, we call enumerate!! and assign numbers, 0, 1, and 2, respectively. The tree pairs of C and H are assigned to 1019 and 1020 which are T(0) = (I)~ - 2 9 1 and T(0) + 1 - (I)7 - 2 9 1 + 1. Similarly the tree pairs of E and H are stored at 1017 and 1018, and the tree pairs of B and H, at 1015 and 1016. We will now present our parallel propagation algorithm. During the propagation, we may have to consider two problem cases caused by redundant pairs. Let a pair (Tri #i) be the newly propagated pair and let another pair (n.i pj) be a pair at a target node of propagation. In the first problem case, a pair (Trj #j) at the target is enclosed by the propagated pair ( ~ #~), i.e., ~ < n.i and pj < #~, then the pair (nj #j) must be replaced by (hi pi). In the second problem case, the pair (Ei p.i) encloses the newly propagated pair ( ~ p~), i.e., r.i < ~ and #i _< #j, and we do not need to propagate the pair (~ #~) to this target. In the propagation, we replace the redundant pairs just described. The results of this algorithm correspond to the results of Agrawal et al.'s algorithm. The boolean function evenp!! returns TRUE on a processor if the processor's ID is an even number. In the

81 A[1

8] D[2 5]

D [2 5]

B

B

[8 8] (7 7)

-.

F [3 5] [71~7] I~, (5 5) ~

G [4 5]

G [4 5] ' ( ~ H [ 5 5]

H[551

(b) Graph After (H, E) is inserted.

(a) OriginalGraph

(c) Before Propagation 0

1

2

3

4

5

6

7

1021 1022 MI-1 MI

A I~I~IBI~I~IGI"I

. . . . . . . 9 ..

. .ll~ ~]ltO-,llE~~1 [~~t r,-,t~4~q~ ~!~


V>

< ( [ 6 7] ~

5

)

. . . . . . .

)

~.,~~~----_,...,~

"'" 1

2

3

4

5

6

7/~

[S811(55)![6711(55,1t8,] /~o15 lO1610171018 lO19lO2OlO21lO22 o . .

Tree Pairs Strand

"~

>

....... (d) After Propagation

o

~

~ Free Space ~

Figure 9. Propagation in Double Strand Representation

Graph Pairs Strand

MI

82 algorithm, the expression redundantii stands for a boolean parallel variable that represents any redundant pairs in the predecessors. As before, in the following functions the expression mark!![x] "- y means that the pvar mark!! on the processor with ID x is assigned the value y. Finding tree predecessors will be different from finding graph predecessors because the tree pairs and the graph pairs are stored in a different form in the tree pairs strand and in the graph pairs strand, respectively. The function target-address!! returns addresses of the target processors of the propagated pairs for tree predecessors and graph predecessors uniformly. Mark-Predecessor(N-Pair, M-Pair" Pair) ; Activate every graph predecessor of a node N which is not predecessor ; of the node M, where N is a new parent node of C and M is the tree ; parent of the child node C. The nodes N and M have the tree pairs N-Pair ; and M-Pair, respectively. Then set the flag mark// on the graph predecessors. ACTIVATE-PROCESSORS-WITH pre!! _!! maxnum(N-Pair)AND!! NOT!!(pre!! __!! prenum(M-Pair) AND!! max!! :>!! maxnum(M-Pair)) DO BEGIN mark!![target-address!!()]:- 1 ; s e t predecessors END Note that, due to propagation, redundant pairs could appear in the marked predecessors. As mentioned before, there are two problem cases caused by redundant pairs. In the first case, the problem could occur only in graph pairs because in this step we are dealing with replacing enclosed pairs with enclosing pairs while in the second case it could occur either in tree pairs or in graph pairs. In the following algorithm, we will present the solution for these problems. For the first case, in the IF!! clause, we examine whether any graph pair in the predecessors is subsumed by the newly propagated pairs but only check the even processors in the graph pairs strand using evenp!! because every graph pair is stored at the even processors in the graph pairs strand. In contrast, for the second case, we examine whether any graph pair and any tree pair in the predecessors is subsuming the newly propagated pair because if that is true, we do not have to propagate the new pair any further. In both cases, the boolean pvar redundantii is set and additionally, in the first case, the enclosed pair is replaced with the number pair to be propagated. Redundant-Pair-Elimination(PM-Pair-V : Pair) ; Replace the pair at the target processor with the newly propagated ; pair PM-pair-V in the first case, set the flag redundant// on ; the target processor in both cases. ACTIVATE-PROCESSORS-WITH mark! ![target-address!! ()] = ! ! 1

83 DO BEGIN ; check whether it is the first case of redundant pairs IF!! (pre!! >!! prenum(PM-Pair-V)AND!! max!! ___!! maxnum(PM-Pair-V) AND!! evenp! !(self-address!! ()) AND!! self-address!!() ___!! g-lb) THEN pre!![self-address!!()]:= prenum(gM-Pair-V) ; replace the prenum max!![self-address!!()]:= maxnum(PM-Pair-V) ; replace the maxnum redundant!![target-address!!()]:= 1 ; set the flag ; check whether it is the second case of redundant pairs ELSE IF!! (pre!! _
At this stage, the boolean pvar mark!! is set for every predecessor of the given node and the boolean pvar redundant!! is set for the processors at which redundant pairs could appear due to the number pair propagation. The next step is to enumerate processors which are predecessors without redundant pairs. Order-Strand( ) ; Enumerate the marked predecessors. No parameter is needed, ; because the global variable mark!! is already set on the predecessors. ACTIVATE-PROCESSORS-WITH mark!![self-address!!()] =!! 1 AND!! NOT!!(redundant!![self-address!!()] =!! 1")AND!! self-address!!()
Now every preliminary step for mapping each predecessor to its corresponding processor ID in the graph pairs strand is finished. Finally, using the functions T ( x ) and T ( x + l ) , the propagation is performed in the following two steps. First, the copies of the tree pairs of the target nodes are copied to their destinations on odd processors. Then the unique pair to be propagated, V, is propagated to the corresponding even processors. Pos!! stands for a parallel variable that contains the numbers 0, 1, 2 ... assigned by enumerate!! in Order-Strand. Propagate-Pair(PM-Pair-V : Pair) ; Propagate U~ pairs to the targets of propagation. The processor IDs ; for the targets are calculated by T(x). ; Propagate the same pair PM-Pair-V to the targets of propagation.

84 ; The processor IDs for the targets are calculated by T ( x ) + 1.

ACTIVATE-PROCESSORS-WITH pos!! >_!! 0 DO BEGIN pre!![g-lb- (pos!! + 1) * 2]:= pre!! ;T(x) max![[g-lb- (pos!! + 1) * 2]:- max[! pre!![g-lb- (pos[! + 1) * 2 + 1]:= prenum(PM-Pair-V) ; T(x + 1) max!![g-lb- (pos!! + 1) * 2 + 1]:= m a x n u m ( P U - P a i r - V ) END Now comes the top level propagation algorithm which combines the above algorithms. It propagates every number pair of a node C to the targets of propagation which were defined by the predecessors of N. The node M is the tree parent of the child node C. Propagation(N, M, C : Node) ; N - P a i r and M - P a i r are the tree pairs of a node N and a node M, respectively. ; P M - P a i r - V is a pair at C to be propagated.

Initialize-Pvars() Mark-Predecessor (N-Pair, M-Pair) ; mark tree predecessors of N FOR Each number pair P M - P a i r - V at C DO Redundant-Pair-Elimination (PM-Pair-V); eliminate any redundant pairs Order-Strand() ; enumerate the marked processors Propagate-Pair(PM-Pair-V)

; propagate U and V pairs ; do some house keeping

Unset-Pvars END FOR END 6. P E R F O R M A N C E

IN THE GR AND

THE DSR

6.1. S p a c e a n d r u n - t i m e c o m p l e x i t i e s We now analyze how many processors are required for implementing the GR and DSR. Agrawal et al. proved that / (N+I)~ 4 ] number pairs are required to represent the worst case of a bipartite graph G with N nodes [18]. Let N be the number of nodes (the number of tree pairs) and P be the number of graph pairs in a DAG; then in the worst case t.

=

4

1

- N.

(a)

Let k be a predefined maximum number of graph pairs for the GR. For the GR the total space requirement is O(k * N). In the worst case k can be O(N) and the space complexity for the GR is O(N2). Note that we are currently fixing k to be 8 as a good compromise between processor use and efficiency of the algorithm. In the DSR the space complexity is O ( N + 2 , G) = O

N + 2,

4

- N

= O ( N 2)

(4)

85 i.e., the same space complexity in the worst case. However, the DSR does not have the problem of unused processors. Now we analyze the run-time complexity for the GR and DSR. Our parallel algorithms for subclass verification and propagation were presented in Sections 4 and 5. In order to analyze the time complexity of these algorithms, we need to define the following parameters: T~(N, C) : Parallel time to determine whether the tree pair of N encloses the tree pair of C. Tg(N, C) : Parallel time to determine whether the predecessors of N have a graph pair from C. Td(N) : Parallel time to determine every predecessor of a node N. Trn(N) : Parallel time to determine every tree predecessor of a node N. T~ (N) : Parallel time to determine every graph predecessor of a node N. Tr (N, C) : Parallel time to replace pairs at the predecessors of N with pairs from C or mark the processors where redundant pairs may have appeared. Tp(N) : Parallel time to propagate a number pair to the marked predecessors. P(C) : Average number of number pairs in C. In the subclass verification algorithm, there are two possible cases with IS-A-VERIFY (N, C). If N is a tree predecessor of C, the run-time for this operation is Tt. If N is a graph predecessor of C, the run-time is 7'9. Assuming a unit communication time [21], a commonly made assumption, Tt and Tg are O(1). Therefore, overall run-time complexity for subclass verification is constant. One question which arises now is whether there are any differences in run-time complexity between the GR and DSR. The difference between the two representations is not in the verification processing, but in the graph pair distribution. The run-time complexity of the subclass verification for the DSR is the same as that in the GR. Consequently, we have a constant subclass verification algorithm in both cases. In the propagation algorithm for the Double Strand Representation, there are two processing steps required as mentioned in Section 5: one for tree predecessors and another for graph predecessors. There are three phases: (1) identify the tree predecessors (Tin) and the graph predecessors (Tn), (2) replace any redundant pairs (T~), and (3) enumerate the predecessors and propagate number pairs (Tp). We can formulate the average run-time for the propagation algorithm as follows:

T --- TIn(N) -t- Tn(N) + P(C) 9 (T,.(N, C) + Tp(N)).

(5)

As before Tin, Tn, Tr, and Tp can be regarded as constants because within constant processor set size, these do not grow with increasing knowledge base size. Then, we can simplify the run-time complexity to O(P). Similarly, the run-time of the propagation algorithm in the Grid Representation is

T' = Td(N) + P(C) 9 (T,.(N, C) + Tp(N)). By the same reasoning, it can be simplified to O(P).

(6)

85

Processor Number

Utilization

of Processors

.vs.

Number

of

Nodes

100.0 ':

'

i

80.0

,?'.

..' ..~, ;.'

-

'

:

i ..,-~:: ..:~.:.-.;

9 9

. :,."

:.,:.,-

9::. . . . . .

;" ,,,..

60.0

i..i -"

!

....

."

4~ ,.~. ,:': ,.. , .-" ..... ..'

40.0

...-.... i i / i ; ~;" i. ..

"!J'

1:3----O GR ,:.:...............:: D S R

!i

' / r-4_~" "

'

"

s - 4 3 - 4~" _ E3 _ E3_I:--I-

0.0

,

0.0

i

500.0

,

,

1000.0 Number of Nodes

,

i

1500.0

E3-

,

2000.0

Figure 10. Processors Utilization

6.2. Experimental results In this section we present experimental results of the parallel subclass verification and number pair propagation algorithms for the GR and DSR. The experiments were done on a Connection Machine CM-5, which makes use of groups of virtual processors executing serially on real processors. There are 32 real processors on the NPAC CM-5. Eve .ry processor emulates the activities of at least 32 virtual processors. The CM-5 [47] is programmed in *LISP, a dialect of Common LISP, by mapping parallel variables (pvar) onto distinct processors.

6.2.1. Experiments with random data For our experiments, we are using a random generator for DAGs. The following parameters are supplied as input to this generator: the number of nodes (N), the branching factor of each node (B), and the depth (D). Prelimina .ry experiments with several values of B and D showed that the computation time seems to be unaffectedby B and D. This should be expected as we have eliminated the explicit representation of the IS-A links from the outset. Therefore, we limited D = 9 , . . . , 12 and set B = 5. The effect of graph size on run-time was determined for both representations. The number of nodes was varied from 25 to 2000. Graphs have approximately 20% of graph arcs, e.g., a graph with 2000 nodes has about 400 graph arcs. For the GR, assume that k is 8. Then 1K processors are required for 1 to 128 nodes, 2K for 1 to 2 5 6 , . . . , 16K for up to 2K nodes. Processor utilization is very low, only up

87 Subclass Verification Run 0.014

Time

.vs. Number

i

of

N o d e s

i

|

[E~-E]- -Ea, l:~:]d'

0.012

--r (1)

,JX~.Fa "Q"

"15]- -IE]- -

r-a----El GR (: .......... -:. D S R

0.010

E I--e-

rr

0.008

i

,

'

]

_

. : ...... ...:-:.:'.L....:..~,.....:.:~..

....

0.006

0.004

. .:7 9 , :-:~:,:.:'::-::. :.,:~......~.,::::.:..--.~ ::.-:. r~

0.0

500.0

1000.0 Number

1500.0

2000.0

of Nodes

Figure 11. Run Time for Subclass Verification

to 18%. We also determined that the maximum number of actually used rows in the GR was 5. Our experiments with random graphs showed that the number of graph pairs increased at approximately the same rate as the number of nodes. For instance, 48 graph pairs are generated in a 100-node graph, ..., 900 graph pairs in a 2000-node graph. In our experiments, typically, the number of graph pairs is limited to less than half the number of tree pairs. According to that, for the DSR, approximately 1K processors are required for graphs with up to 0.5K nodes, 2K processors for graphs with up to 1K nodes, and 4K for up to 2K nodes with very high processor utilization (up to 99%). In Figures 11-13, the run-times jump at two critical points, namely at the node numbers 500 and 1000. These jumps are due to doubling of the number of allocated virtual processors, i.e., from 1K to 2K and 2K to 4K. As the number of real processors stays the same, every real processor has to double the number of operations it performs. The DSR shows better performance than the GR in terms of both the amount and utilization of processors with increasing knowledge base size (Figure 10). For the comparative run-time evaluation of DSR and GR with various sizes of the knowledge base, we implemented the graph insertion, link insertion, and subclass verification algorithms. The test data for link insertion makes a number of simplifying assumptions which are based on problems described in [23]. Figures 11-13 show the results of experiments with various sizes of the knowledge base. The figures show the run times in seconds

88 G r a p h Insertion Run

Time

.vs.

|

0.20

Number

of

Nodes

,

i

G----El

GR

C:.-', ........ ~,-:: D S R

0.15

O

....

G-O-G-E3

(.)

E ._ i-r

0.10

,~:~.

n'--

i

: -,

))

i;: '~.......... ~..:.,...,..:.::....<.-.,,..,~,->

:

rn-D

0.05

/ .: . . . .

[]

...:...-

,..-.,..~"::z".'.::..::,,:..",..:.,2::<, --':. :..,:'. ~: ".:~.:..'::

.... ,...>..

,.-...

t~_~-:~.::.~ .....

0.00

,

0.0

,

,

500.0

i

,

1000.0 Number

of

1500.0

2000.0

Nodes

Figure 12. Run Time for Graph Insertion

over the total numbers of nodes in a graph for the subclass verification (Figure 11), graph insertion (Figure 12), and link insertion (Figure 13) in both representations. As can be seen, the computation times of subclass verification and dynamic update algorithms in the DSR grow much more slowly than in the GR. It is interesting that the execution time for increasing numbers of nodes in a graph are almost the same for constant processor set size. In summary, when we are increasing the size of the graphs, the cost of implementing the subclass verification and number pair propagation algorithms in the DSR in terms of processor utilization is much lower than that in the GR. For run-times, the DSR becomes better for over 500 nodes. 6.2.2. E x p e r i m e n t s with I N T E R M E D Now we want to show experimental results using an existing large medical vocabulary. We have tested our verification and update operations in the Grid Representation and the Double Strand Representation using the INTERMED (INTERnet version of the Medical Entities Dictiona .ry) system of CPMC (Columbia Presbyterian Medical Center) [35-38]. The INTERMED system currently has about 2,500 medical terms. These terms are related by general relations such as IS-A, but also by domain specific relations such as PHARMACEUTIC-COMPONENT-OF. The following results show the necessity of the Double Strand Representation as well as

89

Link Run Time 0.40

Insertion .vs. N u m b e r

'

of Nodes

'

'

r

I 0.35

G----El GR C:.......... ","-) D S R

0.30

0.25

[

~.~ m .E_ 0.20 v-= r 0.15

~

,:

~ ~ ~ :

o. 1 o

0.05

o.OOo.o

-El- -El- -E]- -E3- -El- -E:]- -E~

~.:-:~:.:>.--:::-...... :-:...... ..:.-:....~,::.... ..:.:~....~-!-~...<.:-:..

::::.:

IE]E]IE~"

'

~oo.o

'

~ooo.o

Number

'

~soo.o

....

2000.0

of Nodes

Figure 13. Run Time for Link Insertion without Propagation

the efficiency of the Double Strand Representation. First, we extracted information from the INTERMED and then translated it into a format fit for our system. Each term in the INTERMED is treated as a node and we maintain only the IS-A relation. We used 8K processors with the Double Strand Representation while 32k would be necessary with the Grid Representation. The average run-times for a subclass verification, a graph insertion, and a link insertion in the Double Strand Representation are 0.0004 sec, 0.023 sec, 0.139 sec, respectively. Figure 14 shows the run-time results for the link insertion algorithm over the number of pairs propagated. As the number of number pairs to be propagated increases, the run-time for the link insertion algorithm increases. This confirms our claim in Section 6.1 that the run-time for the number pair propagation algorithm is proportional to the number of number pairs to be propagated. In the Double Strand Representation, 2494 tree pairs and 1442 graph pairs are generated. However, some nodes have up to 456 graph pairs. This means 449 graph pairs cannot be represented in the Grid Representation because the Grid Representation restricts the number of rows to 8. As it would be quite unacceptable to extend the Grid Representation to 512 rows, this result shows that with real data the Grid Representation is not practical at all, while the Double Strand Representation performs well.

90

Link Insertion Run 4.0

Time

of Pairs to b e p r o p a g a t e d

.vs. Number w

|

3.0

r G) tv~ G) E .m pe::3 rr

2.0

J

J

J

J

1.0

0.0 10.0

0.0 The

Number

of Number

20.0 Pairs

to be

30.0 propagated

Figure 14. Run Time for Link Insertion with Number Pair Propagation

7. C O N C L U S I O N In this paper, we have introduced a new massively parallel representation for class hierarchies that are constrained to be representable by directed acyclic graphs. We call it the Double Strand Representation. This representation maps the node set representation onto a linear space of processors. Processor space is divided into two strands, the tree pairs strand and the graph pairs strand. We showed how the DSR can be used to quickly verify the existence of an IS-A relationship that is possibly the result of transitive closure, and therefore not explicitly represented. We also showed how to implement the propagation of a number pair in parallel. Propagation is fundamentally necessa .ry for eve.ry update operation based on Agrawal et al.'s encoding. Experimental results show that the processing times of subclass verification and number pair propagation are mainly affected by the number of allocated processors. The DSR achieves high performance compared with the GR in terms of processor utilization and run-time.

Acknowledgments We thank James Cimino for making the MED available to us and for explaining it to us. Scott C. Pearson has written the random graph generator that was used for providing us

91

with random test data. We also thank Mike Halper who has improved the presentation of this book chapter. REFERENCES

1. R. J. Brachman, On the epistemological status of semantic networks, Associative Networks (N. Findler, ed.), pp. 3-50, New York, NY: Academic Press, 1979. 2. R. J. Brachman and J. Schmolze, An overview of the KL-ONE knowledge representation system, Cognitive Science, vol. 9, no. 2, pp. 171-216, 1985. 3. W . A . Woods, What's in a link. foundations for semantic networks, in Representation and Understanding (D. G. Bobrow and A. M. Collins, eds.), pp. 35-82, New York, NY: Academic Press, 1975. 4. W.A. Woods, Knowledge representation: What's important about it?, in The Knowledge Frontier (N. Cercone and G. McCalla, eds.), pp. 44-79, New York, NY: Springer Verlag, 1987. 5. A. Borgida, R. J. Brachman, D. L. McGuinness, and L. A. Resnick, CLASSIC: A structural data model for objects, in Proceedings of the 1989 A CM SIGMOD International Conference on the Management of Data, appeared as SIGMOD, vol. 18, no. 2, pp. 58-67, 1989. 6. R . J . Brachman, R. E. Fikes, and H. J. Levesque, KRYPTON: A functional approach to knowledge representation, IEEE Computer, vol. 16, no. 10, pp. 67-73, 1983. 7. R. M. MacGregor, A deductive pattern matcher, Seventh National Conference on Artificial Intelligence, pp. 403-408, San Mateo, CA: Morgan Kaufmann, 1988. 8. B. Nebel and K. von Luck, Issues of integration and balancing in hybrid knowledge, GWAI-87 (K. Morik, ed.), pp. 114-123, Berlin, Germany: Springer Verlag, 1987. 9. S. Bayer and M. Vilain, The relation-based knowledge representation of King Kong, SIGART Bulletin, vol. 2, no. 3, pp. 15-21, 1991. 10. W. Kim, N. Ballou, H.-T. Chou, and J. F. Garza, Features of the orion object-oriented database system, in Object-Oriented Concepts, Databases, and Applications (W. Kim and F. H. Lochovsky, eds.), Reading, MA: Addison Wesley, 1989. 11. C. Lecluse, P. Richard, and F. Velez, O2 an object-oriented data model, Readings in Object-Oriented Database Systems (S. B. Zdonik and D. Maier, eds.), pp. 227-236, San Mateo, CA: Morgan Kaufmann, 1990. 12. V. Soloviev, An overview of three commercial object-oriented database management systems: ONTOS, ObjectStore and O2, SIGMOD Record, vol. 21, no. 1, pp. 93-104, 1992. 13. O. J. Dahl and K. Nygaard, SIMULA-an ALGOL-based simulation language, Communications of the A CM, vol. 9, no. 9, pp. 671-678, 1966. 14. A. Goldberg and D. Robson, Smalltalk-80: The language and its implementation. Reading, MA: Addison Wesley, 1983. 15. S. C. Dewhurst and K. T. Stark, Programming in C++. Englewood Cliffs, NJ: Prentice Hall, 1989. 16. L. Cardelli and P. Wegner, On understanding types data abstraction, and polymorphism, A CM Computing Surveys, vol. 17, pp. 471-522, 1985. 17. S. E. Keene, Object-Oriented Programming in COMMON LISP. Reading, MA:

92 Addison-Wesley, 1989. 18. R. Agrawal, A. Borgida, and H. V. Jagadish, Efficient management of transitive relationships in large data and knowledge bases, in Proceedings of the 1989 A CM SIGMOD International Conference on the Management of Data, (Portland, OR), pp. 253-262, 1989. 19. H. V. Jagadish, A compressed transitive closure technique for efficient fixed-point query processing, (San Francisco, CA), pp. 209-223, 1988. 20. K. Guh and C. Yu, Efficient management of materialized generalized transitive closure in centralized and parallel environments, IEEE Transactions on Knowledge and Data Engineering, vol. 4, no. 4, pp. 371-381, 1992. 21. W. Wang, S. Iyengar, and L. M. Patnaik, Memo .ry-based reasoning approach for pattern recognition of binary images, Pattern Recognition, vol. 22, no 5, pp. 505-518, 1989. 22. J. Han and W. Lu, Asynchronous chain recursions, IEEE Transactions on Knowledge and Data Engineering, vol. 1, pp. 185-195, 1989. 23. E. Y. Lee and J. Geller, Representing transitive relationships with parallel node sets, in Proceedings of the IEEE Workshop on Advances in Parallel and Distributed Systems (B. Bhargava, ed.), pp. 140-145, Los Alamitos, CA: IEEE Computer Society Press, 1993. 24. E. Y. Lee, Massively Parallel Reasoning in Transitive Relationship Hierarchies PhD dissertation, NJIT, 1996. 25. J. Geller, Upward-inductive inheritance and constant time downward inheritance in massively parallel knowledge representation, in Proceedings of the Workshop on Parallel Processing for AI at IJCAI 1991, (Sydney, Australia), pp. 63-68, 1991. 26. J. Geller and C. Y. Du, Parallel implementation of a class reasoner, Journal of Experimental and Theoretical Artificial Intelligence, vol. 3, pp. 109-127, 1991. 27. J. Geller, Innovative applications of massive parallelism, A A A I 1993 Spring Symposium Series Reports, AI Magazine, vol. 14, no. 3, p. 36, 1993. 28. J. Geller, Massively parallel knowledge representation, in A A A I Spring Symposium Series Working Notes: Innovative Applications of Massive Parallelism, pp. 90-97, 1993. 29. L. Kanal, V. Kumar, H. Kitano, and C. Suttner, Inheritance operations in massively parallel knowledge representation, Parallel Processing for Artificial Intelligence, pp. 95-113, New York: North Holland Publishing, 1994. 30. H. Kitano and J. Hendler, Advanced update operations in massively parallel knowledge representation, in Massively Parallel Artificial Intelligence, pp. 74-101, AAAI/MIT Press, 1994. 31. L. K. Schubert, M. A. Papalaskaris, and J. Taugher, Accelerating deductive inference: special methods for taxonomies colors and times, in The Knowledge Frontier (N. Cercone and G. McCalla, eds.), pp. 187-220, New York, NY: Springer Verlag, 1987. 32. M. Evett, L. Speetor, and J. Hendler, Knowledge representation on the connection machine, in Supercomputing '89, (Reno, Nevada), pp. 283-293, 1989. 33. M. P. Evett, J. A. Hendler, and W. A. Andersen, Massively parallel support for computationally effective recognition queries, in Proceedings of the Eleventh National

93

Conference on Artificial Intelligence, pp. 297-302, Cambridge, MA: MIT Press, 1993. 34. K. Stoffel and J. Hendler, PARKA on MIND-supercomputers, in IJCAI-95 Workshop Program Working Notes, (Montreal, Quebec), pp. 132-142, 1995. 35. J. J. Cimino, G. Hripcsak, S. B. Johnson, and P. D. Clayton, "Designing an introspective, multipurpose, controlled medical vocabulary," in Proceedings of the Thirteenth Annual Symposium on Computer Applications in Medical Care, pp. 513-518, Los Alamitos, CA: IEEE Computer Society Press, 1989. 36. J. J. Cimino, P. L. Elkin, and G. O. Barnett, "As we may think: The concept space and medical hypertext," Computers and Biomedical Research, vol. 25, pp. 238-263, 1992. 37. J. J. Cimino, A. A. Aguirre, S. B. Johnson, and P. Peng, "Generic queries for meeting clinical information needs," Bulletin of the Medical Library Association, vol. 81, no. 2, pp. 195-206, 1993. 38. J. J. Cimino, P. D. Clayton, G. Hripcsak, and S. B. Johnson, "Knowledge-based approaches to the maintenance of a large controlled medical terminology," Journal of the American Medical Informatics Association, vol. 1, no. 1, pp. 35-50, 1994. 39. M. P. Evett, W. A. Andersen, and J. A. Hendler, Massively parallel support for efficient knowledge representation, in Proc. of the 13th Int. Joint Conference on Artificial Intelligence, pp. 1325-1330, San Mateo, CA: Morgan Kaufmann, 1993. 40. L. Shastri, Default reasoning in semantic networks: a formalization of recognition and inheritance, Artificial Intelligence, vol. 39, no. 3, pp. 283-356, 1989. 41. L. Shastri, Semantic Networks: an Evidential Formalization and its Connectionist Realization, Morgan Kaufmann Publishers, (San Mateo, CA),1988. 42. L. Shastri and V. Ajjanagadde, An optimally efficient limited inference system, in Proceedings of IJCAI-90, (Boston, MA), pp. 563-570, 1990. 43. L. Shastri, A computational model of tractable reasoning- taking inspiration from cognition, in Proc. of the 13th Int. Joint Conference on Artificial Intelligence, pp. 202207, San Mateo, CA: Morgan Kaufmann, 1993. 44. R. Sun, An efficient feature-based connectionist inheritance scheme, IEEE Transactions on SMC, vol. 23, no. 2, 1993. 45. R. Sun, Integrating Neural and Symbolic Processes, Connection Science, 1994. 46. R. Sun, Robust Reasoning: Integrating Rule-Based and Similarity-Based Reasoning, Artificial Intelligence, no. 1, 1995. 47. Thinking Machines Corporation, *LISP Reference Manual Version 5.0 edition. Cambridge, MA: Thinking Machines Corporation, 1988.

94 E u n i c e (Yugyung) Lee Eunice (Yugyung) Lee received a BS degree in Computer Science from the University of Washington at Seattle, in 1990. She is currently a Ph.D. candidate, expecting her Ph.D. in the Summer of 1996, at the New Jersey Institute of Technology. Her research has been published in workshops on parallel AI and on parallel and distributed systems. Her current research interests include massively parallel knowledge representation and reasoning, object-oriented modeling, and high performance distributed databases.

J a m e s Geller James Geller received an Electrical Engineering Diploma from the Technical University Vienna, Austria, in 1979. His M.S. degree (1984) and his Ph.D. degree (1988) in Computer Science were received from the State University of New York at Buffalo. He spent the year before his doctoral defense at the Information Sciences Institute (ISI) of USC in Los Angeles, working with their Intelligent Interfaces group. James Geller received tenure in 1993 and is currently associate professor in the Computer and Information Science Department of the New Jersey Institute of Technology, where he is also Director of the AI & OODB Laboratory. Dr. Geller has published numerous journal and conference papers in a number of areas, including knowledge representation, parallel artificial intelligence, and object-oriented databases. His current research interests concentrate on object-oriented modeling of medical vocabularies, and on massively parallel knowledge representation and reasoning. James Geller was elected SIGART Treasurer in 1995. His Data Structures and Algorithms class is broadcast on New Jersey cable TV. Home Page: http://hertz.njit.edu/-geller

Parallel Processing for Artificial Intelligence 3 J. Geller, H. Kitano and C.B. Suttner (Editors) 1997 Elsevier Science B.V.

95

Parka on MIMD-Supercomputers* Kilian Stoffel, James Hendler and Joel Saltz Computer Science Dept./UM Institute for Advanced Computer Studies University of Maryland, College Park We have ported the SIMD Parka knowledge representation system to generic MIMD machines. The system has been recoded in C and supported using runtime optimization packages developed in the High Performance Systems Software Laboratory at the University of Maryland. New "scanning" algorithms have been developed for inheritance and recognition inferences. These algorithms have been tested with both random networks and on a recoding of the ontology of the CYC knowledge base as well as on large planning case-bases. Tests show that the new version is significantly faster than the SIMD system, and that it promises to scale well to knowledge bases orders of magnitude larger than CYC. 1. I n t r o d u c t i o n In the past, research in the field of (very) large knowledge bases has primarily concentrated on defining KR languages and analyzing their complexity. This research has resulted in a wide range of well-studied languages, with much known about their theoretical performance. Less work, however, has been spent examining details of efficiently implementing such languages, or on the scaling properties of these KR systems using large, real-world, applications. Up until recently, this negligence has largely been justified with the excuse that very large KBs don't yet exist, and that the community could postpone dealing with these issues. However, this excuse is no longer valid. Current research has resulted in the development of extremely large KBs including not only the "common sense" KB of Lenat's CYC [15], but also (much larger) KBs including machine-readable dictionaries [14], large ontologies [9], and very large case-bases for AI planning systems [13]. Despite this, most of the current KR systems are not able to accommodate these KBs, which may contain millions of assertions about many thousands of objects. The ParalleL Understanding Systems (PLUS) Laboratory at the University of Maryland has been working for a number of years on parallel support for very large knowledge bases. The main system to come out of this laboratory was the Parka system, which was the SIMD implementation of a frame-based AI KR language on the CM-2. The system was shown to be extremely fast, and to perform extremely well on very large knowledge bases *This research was supported in part by grants from NSF(IRI-9306580), ONR (N00014-J-91-1451), AFOSR (F49620-93-1-0065), the ARPA/Rome Laboratory Planning Initiative (F30602-93-C-0039, DABT-63-95-C-0037), the ARPA I3 Initiative (N00014-94-10907). Dr. Hendler is also affiliated with the UM Institute for Systems Research (NSF Grant NSF EEC 94-02384).

96 (having on the order of 105 frames). Details of the Parka system and its implementation, as well as a comparison of Parka to other efforts in VLKBs, can be found in [8,6]. There were two main problems with Parka in its SIMD form. First, there was the obvious problem that SIMD computers are currently less popular than MIMD platforms. Second, the ability to scale to much larger knowledge bases than what could be maintained in the memory of the CM2 (i.e. we needed at least one (virtual) processor per node) was constrained by the hardware. This led us to reimplement Parka on a more conventional MIMD platform, also moving from *Lisp 2 to C. In this paper, we describe this reimplementation effort. In particular, the goal of this project is to reimplement Parka for generic MIMD computers. To reach this goal, we use the CHAOS-Libra .ry developed by Dr. Saltz and his students at the High Performance Computing Software Systems Laboratory at the University of Maryland [17]. The language is being implemented in as portable a manner as possible so as to work on a number of different MIMD computers. We will first overview the UM Parka KR project and present some of the applications for which we are developing very large KBs. We then describe in the next two sections Parka's basic data structures and their current implementation. In Section 5 we will analyze the potential parallelism in the system. The central element of the Parka system is the inference engine which is described in Section 6. We present some performance results in Section 7 and we conclude with a summary of the contribution of this work and future research directions. 2. P a r k a O v e r v i e w The Parka system is a frame-based AI language (sometimes called a "property/class" system in today's literature) which was designed to be supported efficiently using highperformance computing techniques. The goal of the project is, and has always been, to develop a fairly traditional AI language/tool that can scale to the extremely large size applications mandated by the needs of today's information technology revolution. More specifically, Parka allows the user to define a frame-based knowledge base with class, subclass, and property links used to encode the ontology. Property values can themselves be frames, or alternatively can be string, numeric values, or specialized data structures (used primarily in the implementation). The language allows exceptions, in the form of multiple-inheritance, and provides extremely efficient (and efficiently parallelizable) algorithms for performing inheritance using a true inferential-distance-ordering calculation [11]. 3 Parka has also been shown to effectively compute recognition, and also to handle extremely complex "structure matching" queries - a class of conjunctive queries relating a set of variables and constraints and unifying these against the larger KB. While it is difficult to exactly compare KR languages, a very loose categorization would put Parka as more expressive than Classic [3], due to the presence of exception handling, 2*Lisp is the parallel version of Lisp which runs on the Thinking Machine Corporation CM-2 and CM-5 multiprocessors. 3Although it has been shown that IDO with fully general exceptions (i.e. cancellation links) is exponentially hard, we've demonstrated that IDO with multiple inheritance exceptions can be computed in polynomial time and is efficiently parallelizable [18].

97 although slightly less expressive than Loom [16] due to the lack of extensive numerical capabilities. A full description of the language, and more details on past results can be found in http://www.cs.umd.edu/projects/plus/Parka. Parka is being used in a number of research initiatives at the University of Maryland. Some of the projects using Parka include 9 A Knowledge Discovery in Databases application for a large medical knowledge base. In particular, it has been noted that data mining systems can benefit from the use of KBs to provide semantic information for data mining [4]. We are exploiting this sort of information by using medical knowledge in the data mining of an OB-Gyn patient database containing records on over 20,000 patients. The current Parka KB has over 1.2 million assertions [20].4 9 Several Case-based planning applications including the memory-intensive case-based planning system, Caper, developed by Brian Kettler in his recent doctoral thesis [13]. The Caper KBs, the largest of which contains over 2,000,000 assertions, are described later in this paper (section 4). Parka is also being used to support a logistics planning project, jointly being developed with MITRE Corp. [10], with a KB currently containing over 650,000 assertions. 9 A recent project involves the use of Parka as a hybrid knowledge and database for storing and retrieving designs and process plans for mechanical products, as part of ongoing effort to develop a new hybrid variant/generative approach to process planning in manufacturing. Parka's content-based retrieval of designs and plans is important in order to achieve the desired functionality of the process planning system, and will represent a significant advance over current "variant" approaches to process planning, which (among other things) involves retrieving process plans from databases based on fixedlength alphanumeric keys. The KB for this project is currently under construction, but is expected to dwarf those described previously, since current part databases are extremely large, containing information about the machining of thousands of parts. 3. Parka~s basic d a t a s t r u c t u r e s The need for portability has influenced the choice of the environments to implement Parka: C o m p i l e r : The compiler used was a standard ANSI C one. This compiler exists on all of the computers we currently access, particularly including the IBM SP-2 and the CRAY T3D. C o m m u n i c a t i o n L i b r a r y : Our implementation is based on functionality provided by the CHAOS run-time support package which provides scheduling libraries that have been ported to a wide range of MIMD supercomputers. (Work on the Parka project has actually led to a new scheduling approach which is being integrated into the CHAOS libraries[19]). 4We report KB sizes in "assertions," basically the number of links in the semantic network corresponding to the frames, as this accurately reflects the total number of relations between items in the KB and corresponds directly to the "concepts" in a description logic representation.

98 We have focused on the design and implementation of a very general system that can easily be ported to the different supercomputers. This means that currently, no special optimizations are done using the specific characteristics of any one machine. (Future work will include optimizing on several important architectures.) 3.1. T h e d a t a s t r u c t u r e s of t h e K B To use CHAOS and other generic tools, it was necessary to reimplement the Parka system to be based on arrays, rather than on the linked lists of the earlier *Lisp implementation. There are three major structures used: F r a m e s : The primary data structure used in the new implementation is a frame. A frame is an array of a variable size. Each element of the array points to an element of the Network Array described below. This pointer defines the properties of the frame. If an element points to a "normal" frame then the element represents an ISA-link. If the pointer goes to a frame that is characterized to be a property, then the link defines an explicitly valued property of the frame. This semantic difference allows us to separate the links into two sub-arrays, one for ISAs and one for properties.

A property frame (property) has a different structure than a normal frame. The basic structure of a property frame is again an array. This array contains two subarrays of the same size. For each non-ISA property in every normal frame, there is a pointer to a property frame. In the property frame, there are two links which are created, each in one of the sub-arrays. The link created in the first sub-array points back to the frame for which the property was defined. The second link points to the value for the property of that frame. In addition, a property frame also contains the arrays of a normal frame (see Figure 1). (This allows properties themselves to be frames with hierarchies and other properties, which was not allowed in the earlier SIMD system.) N e t w o r k - A r r a y : The network array is an array which holds all of the normal and property frames in the system as sub-arrays. (This array can be quite large for very large KBs.) F r a m e - T a b l e : The frame table is an array of strings. In each element of the frame table the name of a frame is stored. The name corresponds to the name of the frame located at the same position in the Network Array (e.g. the fifth element of the frame table contains the name of the fifth element of the network). This allows a mapping from "human" to machine descriptors, for use in querying, etc. This representation of the KB has several considerable advantages. 9 The representation of the KB as an array (of arrays) allows us to use standard communication libraries like CHAOS. In addition, computation over array-based data structures is a well-studied area for MIMD Systems. 9 The separation of the properties from the ISA links allows a fast parallel implementation for inheritance algorithms.

99

Frame

I I ISAS I I I I I~1 II portiesII

~

Property

I

II

I

FROM

II

I I I

I I I II ISAS

Frame

I

I I Properties I I I I I

IIIIIIISAS I I Properties I I II

'

I1

I

Figure 1. Frame Arrays in MIMD Parka

9 The replacement of the single property link by a frame permits efficient implementations of different parallel algorithms such as recognition and structure matching. 9 A separate name space (Frame Table) can be used to calculate starting points for scan operations without the presence of the whole KB.

4. C u r r e n t S t a t u s

We have currently reimplemented all of the major algorithms of the Parka system the creation of frame-based semantic networks, the performance of inheritance operations thereon, the execution of recognition queries and structure matching queries, all of which we explain in the following sections. 4.1. C o n s t r u c t i o n of t h e F r a m e s , t h e N e t w o r k a n d t h e F r a m e T a b l e There are three methods for creating and/or loading the networks on a parallel machine. 1. The KB is defined directly by a program using commands such as define_frame, define_property, define_isa and assert. 2. A KB can be read in from an ASCII file containing these same commands. 3. If a KB is once constructed then the memory can be dumped on the disk. The dump files can be read in again later. It is possible to generate either a dump of the whole network or a distributed dump for multiprocessors. In the latter case, a separate dump for each node is created. All three methods can be used to generate one network on a single processor or a network being distributed over the nodes of the parallel computer. There are two predefined

100 methods that distribute the network in blocks or stripes. (It is also possible to use other methods to distribute the network which are predefined or user-defined.) In the first two methods (where a network is created) a consistency check for the whole network is performed. For the third method it is assumed that the network is correct. 4.2. Inheritance For the efficient performance of inheritance, we require a fast algorithm for scanning the ISA hierarchy. The efficiency of this algorithm is crucial to supporting very large knowledge bases because, as we argue in [8], most interesting inferences in large knowledge bases are dependent on the efficiency (and correctness) of the inheritance algorithm. 4.2.1. The Scanning A l g o r i t h m The scanning algorithm is dependent on the distribution of the Network Array among the processors. The current implementation supports the two predefined types (block and cyclic) of distributions. The user is free to define other distributions, but all of these distributions must be static. In principle, there are two approaches as to how the scan algorithm could work: 1. To expand the ISA hierarchy of a frame, a processor has to find out on which node the frame is located. If the frame is on the local node the ISA hierarchy can directly be expanded. If the frame is installed on a remote processor, then the local processor could ask for a copy of the frame from the remote processor and could then expand the hierarchy. This is essentially a "load balancing" solution, as frame information is propagated amongst processors. 2. The local processor has again to locate the frame in question and expand it directly if its local. Otherwise it will inform the remote processor to expand the corresponding frame. This approach (which we use) has more of a scheduling burden (coordinating communication) but needs less explicit load-balancing. In the second approach, the work load is dependent on the distribution of the Network Array. Each frame is expanded on the processor which hosts that frame. The first approach does not contain such an implicit distribution of the work load. Instead, it would require that an explicit distribution of the work must be defined. However, the amount of work to realize during the expansion of a frame is very small (only a few instructions) compared with the communication overhead. It would be very hard to implement an efficient load balancer because of the supplementary network load produced by the balancer. For this reason we chose the second approach for our implementation. Each frame is expanded by the processor on which it is stored. This leads to the following basic scan algorithm as used in this implementation: I. locate the starting frame 2. look up the ISA array of the frame 3. send msgs to all processors which host a frame defined in the ISA array 4. receive all msgs from remote processors

I01

5. f o r each r e c e i v e d frame goto s t e p 2 This program is executed in SPMD mode on each processor of the parallel computer. Based on this scan algorithm it is now possible to define an inheritance algorithm.

4.2.2. Implementation of the Scanning Algorithm Our programming model attempts to have as much of a data-parallel flavor as possible. This places restrictions on the types of task-parallelism we allow. We list these restrictions below. 1. The task-graphs (network array) are static and acyclic. 2. Each data item (frame) is updated by a unique task and all data-items updated by a given task are on the same processor. This allows an owner-computes strategy. 3. The same task-graph is repetitively used for different data-sets. This allows runtime preprocessing optimizations. These restrictions are not severe and the previously described implementation of the Parka system satisfies these restrictions. The nodes for the task-graph are distributed across processors based on a user-specified distribution and the computation is distributed using the owner-computes rule. Since the task-graph computation is iterative, certain optimizations can be performed once in a preprocessing step and reused. One such optimization is to perform a topological sort of the DAG, thus dividing it into levels. As shown in Figure 2a, the synchronization requirements can now be confined to the levels thus increasing the computation granularity. This method has been suggested in [17], and was used to parallelize sparse matrix codes with loop carried dependencies. However global synchronization does not work ve.ry well because the synchronization constraints affect the load balancing. All processors have to wait for the slowest processor at each level. The total computation time is __~. #/=processors Comp (procj, leveli) ) 5 thus z~i=v'#teve~sl(,,,a~j=l The alternative approach, considered in [5] is to use better distribution mechanisms to increase locality and then use low-latency active messages to communicate the data that does need to go off-processor. The arrival of data automatically triggers computation, thus the synchronization is implicit. The synchronization requirements of this method are ve.ry relaxed and thus the load-balance is better. The total computation time is #processors [W-,#levels Comp (procj, level~)). While the computation time is better than max~=l the level-synchronized scheme, the communication/synchronization time may be worse since more messages are sent, even though the amount of data sent is the same. [5] found that high efficiencies could be achieved on the CM-5 using its very low overhead communication support. A drawback is that a data flow programming model must be used. Our approach is to use the level-based data-parallel model but relax the synchronization constraints by using split-phase synchronization mechanisms. This allows processors to continue processing incoming data from the next level while waiting for the slowest 5Where Comp(proci,leveli) is the time used by processor j to compute level i.

102

DO K = 1, num_levels MPI_scatter(level(k-I)) MPI_gather(level(k)) COMPUTE(level(k)) END DO

a) Collective synchronization

DO K = 1, num_levels id = FZY_scatter(level(k-I)) WHILE ( FZY_recv(id, data) LEVEL_NOT.DONE)) COMPUTE(level(k)) END DO

b) Fuzzy synchronization

Figure 2. Collective/Fuzzy synchronization

processor from the previous level. Since each processor does not progresses to the next level until it has processed all nodes at the current level, the skew is limited to a maximum of one level. This is most useful when a processor can have a high load at one level and a low load at the next level. In such a case, that processor can catch up without slowing the others down. Figure 2b shows how a DAG could be parallelized using such a fuzzy barrier. Each processor sends out all the outgoing data at the end of each level, but processes each incoming data-chunk as soon it arrives. The condition NOT__DONE remains true until all the incoming messages from a level have arrived, or when a termination condition has been detected. Thus even if processor A is on level X, processor B can begin to process data on level X + I (until it eventually waits to receive A's message). Though, we do not discuss it here, the termination detection check can be incorporated to implement branchand-bound algorithms. The computation time is thus better than that of the level-based scheme with no extra communication overhead. Figures 3 and 4 shows the results of using fuzzy synchronization and the level-based runtime techniques on the inheritance network. These experiments used an inheritance network with 500,000 classes and instances (i.e. the DAG has 500K nodes) and several links between each (for a total of about 1.5M links). As can be seen from Figure 3, fuzzy synchronization provides significant benefits over using collective synchronization. Figure 4 shows that high efficiencies can be obtained over a wide range of platforms using these techniques. The efficiencies are around 70%. 4.2.3. T h e I n h e r i t a n c e A l g o r i t h m The inheritance algorithm we use is based on the Touretzky IDO algorithm as modified in Parka. The theoretical worst case complexity of this algorithm is O(d 9 n/k) where d is the depth of the ISA-hierarchy, n is the number of frames in the network and k is the number of nodes of the parallel computer. The idea of our algorithm is that in one pass through the ISA-hierarchy, we can simultaneously create a list of all properties (direct and inherited) for each node and also a list of all properties "overwritten" by more specific properties (therefore being eliminated from the first list). The difference of these two lists contains the properties inherited according to IDO. We discuss the efficiency of this algorithm in section 7.2.2 (Table 1).

103 Barrier vs. Fuzzy Synchronization on SP-2

.

.

.

.

.

.

mo, Fu~,-':4--.

|

Ide'~,l -m--. 1

...

..- ..

.o

9-'"'"'"'"

0 0

!

!

2

4

i .... "" . . . . ~....

!

6

8

10

12

14

16

#nodes

Figure 3. Barrier vs. Fuzzy Synchronization

5. P o t e n t i a l P a r a l l e l i s m in K B

In this section, we discuss the kind and degree of MIMD-parallelism we exploit. The parallelism of a KB is hidden in the ontology. In our case the ontology is represented as a directed acyclic graph. In such a graph the parallelism usable by the scanning algorithm is defined by the branch out of each frame in the network (i.e. the number of ISA links per frame). A network in which each frame has only one ISA link defined has no parallelism exploitable by the scanning algorithm. Each frame has to be completed before the descendent frame can be attacked. Such a graph imposes serial execution of the scanning algorithm. On the other hand, a graph with depth two and a large amount of ISAs defined for the root frame has the highest possible degree of parallelism for the scanning algorithm. This algorithm therefore seems to be well adapted for AI KBs. Generally, these are not very deep but often very bushy. The fact that the parallelism is low for narrow ISA hierarchies is not very harmful. In addition, in several types of queries there are multiple scans to perform. These scans can be combined and executed as one single larger scan. Such a method is used to implement recognition queries (see Section 6.1) and in more complex inference methods like the structure marcher (see section 6.2).

6. C o m p l e x I n f e r e n c i n g in M I M D P a r k a As mentioned previously, we have built several other inference classes on top of the inheritance mechanism. In this section, we discuss the two most important: recognition queries and structure matching inferences.

104 ,

i

i

sp2

sp2/mpi -4--.

x.,..~, *'~-..~

4

2

u

1

"

0.5

Paragon -I~--.

~"-.-......

cm5 -~(,.-t3d/pvm -,&---

-.~.,.~.. ~'-...~.,..

"'-.,..... ""'~''" "1~.,..~..,.

9

"~'~"~''~.~.

99~-..~.~._.,.

".-..~.,..

........

-......:::...., .

025

0.125

0.0625

i 2

!

!

4

8

#nodes

Figure 4. Performance of parallelized Inheritance Network code: Timings shown include the native communication systems for the SP2, the Paragon and the CM5 use the native communication system as well as MPI and PVM times for the SP2 and T3D machines respectively.

6.1. Recognition We have implemented a new version of Parka's recognition algorithm - its means for handling conjunctive property queries. 6 The algorithm is based on the special structure of the property frames and again uses the scanning algorithm. Essentially, we handle these queries as a set of inheritance queries and some join operations, thus we can maintain the complexity of O(d 9 n/k). To process the conjunctive property query, we split the conjunction into the conjunctive elements. This produces a list of simple expressions of the form (prop1, x, y) (where x and y can both be variables, or one of the two can be a constant). So we have to find all the frames which have an explicit or an inherited definition for the given property and their value in respect to the property. It is easy to find all explicitly labeled frames for a given property. The sub-array of the property frame with the back pointers contains all the pointers to the frames which are directly labeled for this property. The other sub-array of the property frame holds the pointers to the value for the property. On one hand, it is possible to create a list with all the frames for which the property is directly defined and which have the searched value. On the other hand, a list with all the frames which do not have the value being searched for, but which are able to be "overwritten" with the correct values during the inheritance can also be created. The basic idea of the recognition algorithm is that all the frames which have the required property value are activated and start to send information downward along the ISA links. 6The SIMD algorithm for performing this was one of the main contributions of the original Parka system[7].

105 During this scan, all frames in the ISA hierarchy will inherit the values of the property. If the inherited information arrives on a node which is directly labeled for a property, but with a different value, then all the descendants of this node are commanded to remove the value they inherited from the same node as this node. Thus, it is possible to correctly handle all inherited properties according to IDO in one single scan. We can find the solution to each sub-query defined by the elements of the original conjunctive property query in one single IDO run (scan run). The result of this run is a number of sets containing the frames and their values for the analyzed property. Frames which are elements of all the sets and their corresponding values are the solutions of the original conjunctive query. T The potential degree of parallelism of this algorithm is high, because there are multiple starting points (all the frames with direct correct property values). At the same time it is possible to send multiple properties across the ISA hierarchy during one scan step which results in a large scan graph with a high degree of parallelism.

6.2. Structure Matcher The second even more complex inference method is the structure matching algorithm. It allows general relation-based structures to be retrieved from a knowledge base. The algorithm takes a conjunctive query similar in form to the recognition query, but with binary constraints (i.e. with two variables). A set of these conjuncts defines a graph structure, where variables or constants are the nodes and predicates are the arcs. The algorithm compares the probe structure with the knowledge base and returns a set of all satisfying matches. Our description of the problem of structure matching follows that given in [21]. A knowledge base defines a set, P, of unary and binary predicates. Unary predicates have the form Pi(x) and binary predicates have the form P.~(xl,x2), where each x~ is a variable on the set of frames in the KB. An existential conjunctive expression on these predicates is a formula of the form 3 x l , . . . ,Xm " 1 ' 1 A P 2 A , . . . , APn, where n >_ 1. Our task is to retrieve all structures from memory which match a given conjunctive expression. Therefore, we would like to find all such satisfying assignments for the x~. We can view the problem of matching knowledge structures in two ways. The first is as a subgraph isomorphism problem, s We view variables as nodes and binary predicates as edges in a graph. We want to find structures in memory which "line up" with the graph structure of the query expression. The other way to view the matching problem is as a problem of unification or constraint satisfaction. If we can find a structure in memory which provides a consistent assignment to the variables xi (i.e., unification), then that structure matches the conjunctive expression. 6.3. O v e r v i e w of the algorithm The structure matching algorithm operates by comparing a retrieval probe, P, against a knowledge base (KB) to find all structures in the KB which are consistent with P. This match process occurs in parallel across the entire knowledge base. A Parka KB 7This can be formalized as a set of joins in the relational calculus. SMore specifically, this is a problem of subgraph isomorphism with typed edges, the edges being the relations in the KB between frames.

106 consists of a set of frames and a set of relations (defined by predicates) on those frames. Most relations are only implicitly specified and so must be made explicit by expanding the relation with the appropriate inference method. By computing inherited values for a relation, all pairs defining the relation are made explicit. We currently allow only una .ry and binary relations. 9 A retrieval probe is specified as a graph consisting of a set of variables V(P) and a set of predicates (or constraints) C(P) that must simultaneously hold on frames bound to those variables. The result of the algorithm is a set of k-tuples, where each k-tuple encodes a unique 1 - 1 mapping of frames to variables in V(P), that unifies the description of the structure in memory with C(P). Figure 5 shows a simple example of the structure matching algorithm; given a knowledge base represented as a semantic network and a probe graph, the algorithm finds all matching subgraphs in the network which are isomorphic to the probe.

Probegraph

Fragmentof semanticnetwork color

(~

~

color

jj

color

I

StructureMarcher

/ I

Matching subgraphs

Figure 5. A simple example of structure matching.

The set of frames that can bind to each variable is initially restricted by a set of constraints indicated by unary predicates. Each una .ry constraint may only constrain the gFuture versions of Parka may support higher-order predicates.

107 values of one variable. Examples of these constraints are "X is a dog" or "the color of X is yellow". We allow set theoretic combinations of the,unary constraints; for example "X is a dog and the color of X is yellow", or "X is a dog but X is not yellow." The domains for each variable are maintained throughout the match process and are further restricted as more constraints are processed. Constraints between frames bound to variables are specified by a set of binary constraints. For example, we can say "the color of X must be Y", or "X is a part of Y", for some X and Y in V(P). Binary constraints are processed by "expanding" the binary relation given in the constraint. By expansion we mean that all pairs participating in a relation R in the KB are made explicit by invoking the inference method for the associated predicate. The pairs allowed to participate in the expanded relation are restricted to those in the domains of the variables related by R. For example, a binary constraint may be expressed as (Color X Y). In this case the values for each concept in the domain of X are computed for the color predicate and pairs that have values outside the domain of Y are excluded. Two additional binary predicates, "eq" and "neq," are provided to provide co-designation and non-co-designation of variables. These constraints act as a filter, eliminating any tuples from the result for which the constrained variables are (not) bound to the same frame. (Other binary predicates could be supported as well, such as inequality and other numeric constraints. However, in the current implementation, we do not provide support for these constraints.) The result of a structure match is a set of k-tuples, where each tuple corresponds to a satisfying assignment of the k variables. Alternatively, the result can be viewed as a relation. Initially, the matcher begins with an empty set of relations. During the match, several intermediate relations may be constructed. Simple binary relations result from the expansion of a binary constraint. These are later fused (via a relational join operation) or filtered (via co-designation or non-co-designation) until a single relation remains. The algorithm selects binary constraints to process using a greedy algorithm based on a simple cost model stored in the meta-data. 7. Testing To realize the tests it was necessary first to program a network generator. The networks generated by this program were afterwards used to test the implemented KB.

7.1. Network generator The network generator is able to generate two kinds of KB formats: 1. ASCII file 2. memory dump files There are three characteristics of the network that can be defined: 1. the branching factor in the network 2. the number of levels

108 3. the number of levels with the maximal size Further it is possible to randomly generate properties for different frames. There is a parameter to define the density of these properties. This allows us to duplicate some of the tests made on the original Parka system on the SIMD CM-2 computer (cf. [8]). 7.2. Results on I B M S P 2 (16 processors)

7.2.1. Degree of Parallelism As mentioned above, the performance of the scan algorithm depends on the structure of the ISA hierarchy. To test this, we created two networks, both with 2,500 frames. The first network is 2, 500 levels deep and each level consists of one frame. The second network has two levels, a first where one frame is inserted and and a second with 2, 499 frames. For the sequential algorithm the amount of work is the same for both networks. This is also true for the parallel scanning algorithm but the big difference is that for the first network there is little opportunity for parallelism - one level has to be executed after another. In the second network it is different. All the frames on the second level can be treated in parallel if they are distributed among the processors. To execute this first test, we used 4 processors on an IBM SP2. A cyclic network distribution was used. Thus, processor 1 owned the frames {0, 4, 8, ...}, processor 2 owned the frames {1, 5,9, ...}, etc. The execution of the scan algorithm on the first network took 0.4 s. The scan through the second network only 0.008 s. Note that although the theoretical speedup should be no more than 4, we have a speedup of 50! This is related to the communication system of the SP2 (and most other MIMD machines). For the first network we have to communicate 2499 short messages, in the second one long message. The throughput of the communication system is much higher on a small number of long messages then on a large number of short messages. This result seems to be very promising for real KBs, because most of the existing KBs have a very flat ontology. For example, in a recoding of the ontology of CYC for Parka (see section 7.2.3), the deepest subgraph we found was only 27 levels deep, whereas maximum branch out was over 1,000.

7.2.2. Scaling and Speedup As a second test, we want to analyze how the scanning algorithm performs if the configuration of the computer changes. The results presented here are again based on generated networks. The branching factor was fixed to 3. The depth of the ISA-hierarchy was varied from 9 to 12, generating networks of about 29,500, 88,500, 265,700, and 797,000 frames respectively. (As a point of comparison, our encoding of the ontology of CYC (Section 7.2.3) has about 32, 500 frames.) A property to find was placed on the root node and the inheritance was done by one of the leaf nodes (this corresponds to the worst case for the previous Parka implementation). The absolute values represented in Figure 6 and Table 1 are extremely fast, however we note that they are not so important because the programs were not optimized for the SP2 where the tests were done. What is interesting is the comportment of the algorithms. There are two points that seem to be interesting: 1. the scalability in terms of processor number

109

nbr. nodes/netsize .... 1 0.115 29524 0.345 88573 1.037 265720 797161

2 0.060 0.175 0.523 1.689

4 0.033 0.091 0.265 0.735

8 0.021 0.050 0.137 0.388

16 0.017 0.032 0.088 0.201

Table 1 Inheritance

Scanning-Performance using CYCLIC-distribution 1.6

-

u

1 0.8 ~ 0.6 [ __-.......... 0.4 time [sec]

0.2

--

""

i

""'"

,

"''"'.

~

i

l

,

,

,

,

,

i,

i

,-

29524 ~ 88573 .-F-- ~<. . . . . . . 265720 4~- .... "" x 797161 .x.-

"......+..

" ~ " 9 ,...)

<

1

Figure 6. Scanning Performance

I

I

I

2

3

4 5 nbr. nodes

I

I

I

6

7 8

16

110 2. the scalability in terms of network size

On both measures, we see that the algorithms perform quite well. In addition, we believe further speedup would be available if we optimized the algorithms for this particular platform.

7.2.3. Results for the C Y C Ontology We have performed a number of tests on KBs that were not randomly created. The largest one we had used previously in testing the SIMD version of the system was the CYC ontology. As previously reported in [7], we encoded the ontology of version 8 of MCC's CYC system into Parka. In our version, this ontology has about 32,000 frames and about 150,000 assertions relating them (property and ISA links). The biggest difference between the generated networks and CYC is that the test network had only a single property on the root, thus making all "subgraphs" equal to the entire KB. In CYC the ISA hierarchies under particular properties are only subnetworks of the full 32,000 frames. We hoped to directly compare the SIMD and MIMD versions using queries on the CYC system, but most of the tests we had run previously were, basically, too easy for the new version. In particular, the subnetworks in CYC are all much smaller than the smallest test network described in the previous section. For this reason simple queries like "What is the color of frame X" could not be used to show parallel effects - their time was on the order of 50-100 #seconds on a single processor! Thus, instead of timing single inheritance queries, we generated a new class of recognition-like queries that were designed to stress the new system. To start with, we tried the timing of recognition queries in some of CYC's largest subtrees. The results were quite promising, but again too fast for showing parallel speedups. For example, using the scan algorithms, we executed some recognitions with more than twenty conjuncts and had single processor response times of under 1/100 of a second (compared with about 1 second for the SIMD system). To further stress the scanning algorithms and to explore speedups, we designed a new set of inferences specifically to that purpose. In particular, we used queries that would include great amounts of both inheritance and recognition. These queries were of the following form: "Give me all the frames which have one or more properties in common with frame X." To be sure to get "slow" times, we ran these queries on frames in big ISA sub-hierarchies of CYC. Two of the biggest, plant and animal, were chosen for testing, since they had very large numbers (in the tens of thousands) of other frames which shared at least one property. The execution time for the queries "Give me all the frames with the same properties as Animal" and "Give me all the frames with the same properties as Plant" are presented in Figure 7, which compares these times to the optimal speedup curve. As one can see the recognition algorithm behaves very well. The efficiency is about 75%. That is very high with respect to the small amount of CPU time needed for the inheritance algorithm. The queries represented in the Figure 7 are much more complex than standard queries in CYC.

111

Inheritance-Performance in CYC 8

I

I

I

t~..

Animal ~-Plant + - _

4 2 time [sec] 1 0.5 0.25

1

I

I

I

2

4 nbr. nodes

8

16

Figure 7. Recognition-Performance in CYC

7.3. R e s u l t s for U M - T r a n s l o g A second domain used for testing the MIMD implementation was on knowledge bases that were created as part of doctoral thesis work on the development of a system for memory intensive case-based planning [12]. Part of this project involved the automatic seeding of large case memories by a generative planner. One domain used in this work was the "UM Translog" domain, a logistics planning domain developed for the evaluation and comparison of AI planning systems. Case-bases of various size were created, with each containing a number of plans, derivation information, and planning related ontologies. 1~ To measure the performance of the structure matcher we used the UM-Translog KB in different sizes (10 cases, 100 cases and 200 cases). The results presented in this section are especially interesting for two reasons. On one hand we are presenting the timings for the structure matcher, the most complex retrieval algorithm implemented in Parka to date. On the other hand we are also able to show, that the new Parka system has the capability to handle very large KBs on a single Sparc workstation. The queries we used to test the system are presented in detail in Appendix A. These are all queries used by the case-based planning system during its problem solving. They were chosen at random from a large number of stored queries. The results are summarized in Table 2. Table 2a presents the timings for the six queries on a SPARC 20 using the three different KBs, and Table 2b presents the timings on 1, 8 and 16 nodes of an SP2 using the 200 case KB (the largest of the three). The actual sizes of these KBs are shown l~ full description of the domain, the complete specification of the planning operators used, and the casebases themselves are available on the World-Wide Web at http://www.cs.umd.edu/projects/plus/UMT/.

112

II Query 1 2 3 4 5 6

20 CB 1020 195 225 630 675 405

100 CB 4740 1305 1470 3570 3600 585

200 CB II II Query 6990 1635 1725 4590 4605 645

a: Timings on a SPARC 20 for a 20, 100 and 200 cases KB

1 3041 713 753 1997 2003 284

8 546 129 135 361 360 52

16 313 75 79 205 206 29

1:8 69.6 69.1 70.0 69.1 69.5 68.2

1:16 II 60.7 59.4 56.6 60.9 60.8 61.2

b: Timings on a 1, 8 and 16 nodes of an SP2 on 200 case KB

Table 2 UM-Translog Timings[msec] (serial and parallel).

II cases I frames I structural links 20 11612 26412 100 59915 130558 200 123173 266176

property links II 114359 800481 1620456

Table 3 Sizes of the UM-Translog KBs.

in Table 3, where frames is the number of nodes in the DAG, structural links are those in the ISA ontology, and property links are all others (i.e. the number of the edges in the DAG is equal to the structural links plus the property links). (We believe the 200 case case-base to be the largest meaningful semantic net created to date. It contains about 1.8M assertions concerning over 123,000 entities.) As can be seen, the sequential timings range from under a second for the simplest query to about 7 seconds for the most complex. On the parallel system all queries were executed in under one second, with the simplest query taking only 29 milliseconds on 16 processors, and the most complex taking only 303 millisecond. Table 2b also shows the efficiencies of the parallel algorithm. Even in the current non-optimized form, the efficiency averages about 69.3% for eight processors and 59.9% for 16 processors. 8. F u t u r e work We are currently working on pushing the current implementation and developing new algorithms and applications. Algorithmically, current work is focused on doing empirical research on how these algorithms scale to much larger networks and greater numbers of processors. In particular, the current implementation is done on a small IBM SP2 machine (16 "wide" nodes). The next step will be to test these programs on bigger SP2s to see how the scaling continues in larger networks and on larger machines. We also will do work on optimizing communication patterns for the SP2 implementation (which should

113 significantly improve absolute performance) as well as porting this implementation to the Cray T3D machine, which has significantly faster communication times. We expect to be able to run extremely large networks (potentially billions of frames) extremely efficiently on this machine. First tests of the basic scan algorithm seem to support this assumption. In addition, we are exploring the use of I/O to allow the use of secondary storage for processing extremely large knowledge bases where even the online memory of the large parallel machines is not adequate. The CHAOS runtime support package is being extended to support efficient I/O, and we are also beginning an examination of the special I/O requirements for the storage and caching needs of much larger knowledge bases (containing 100s of millions of assertions). The SP2 at Maryland is being configured to contain about 200 Gigabytes of disk storage, making it uniquely suited for exploring massive knowledge and database applications. In regard to applications, we are currently exploring the use of knowledge based techniques in the guidance of data mining applications. In data mining applications, a knowledge repository is searched for information with the aim of obtaining as much information as possible. One weakness with current data mining systems, however, is their lack of access to semantic knowledge about the domain being searched. Storing such semantics, however, requires much larger knowledge bases that function significantly faster than most of today's systems. Such systems will be needed for doing significant knowledge discovery against large, heterogeneous data corpora such as medical data systems or the World Wide Web. Parka promises to provide support for such knowledge bases. We are currently exploring the development of such hybrid knowledge and databases in the areas of medical informatics, military transportation logistics, and health care financing. 9. C o n c l u s i o n In summary, we have ported the SIMD Parka system to more generic MIMD machines. The system has been recoded in C and supported using runtime optimization packages developed in the high performance computing laboratory at Maryland. New "scanning" algorithms have been developed for inheritance, recognition, and structure matching inferences. These algorithms have been tested with both random networks and on two very large knowledge bases (CYC and the UM-translog case bases). Tests show that the new version is significantly faster than the SIMD system, and that it promises to scale well to knowledge bases of the order of magnitude needed for complex information technology applications. REFERENCES

1. Proceedings of the 12th National Conference of the American Association for Artificial Intelligence. AAAI Press, MIT Press, 1994. 2. W. Andersen, , M. Evett, J. Hendler, and B. Kettler. Massively Parallel Artificial Intelligence, chapter Massively Parallel Matching of Knowledge Structures. AAAI/MIT Press, 1994. 3. A. Borgida and P.F. Patel-Schneider. A semantics and complete algorithm for subsumption in the classic description logic. Journal of Artificial Intelligence Research, 1, 1994.

114

10.

11.

12. 13. 14.

Paolo Bresciani. The challenge of integrating knowledge representation and databases. In KARP-95 Second International Symposium on Knowledge Acquisition, Representation and Processing, pages 16-18. Auburn University, AL, USA, 1995. Extended version in IRST TR 9412-06, IRST, Povo TN (Italy). Fredric T. Chong, Shamik D. Sharma, Eric Brewer, and Joel Saltz. Multiprocessor runtime support for fine-grained, irregular DAGs. Technical Report CS-TR-3266, University of Maryland, Mar 1994. To appear in Parallel Processing Letters. M.P. Evett. PARKA: A System for Massively Parallel Knowledge Representation. PhD thesis, Dept. of Computer Science, University of Maryland, College Park, 1994. M.P. Evett, J.A. Hendler, and W.A. Andersen. Massively parallel support for computationally effective recognition queries. In Proc. Eleventh National Conference on Artificial Intelligence. 1993. M.P. Evett, J.A. Hendler, and L. Spector. Parallel knowledge representation on the connection machine. Journal of Parallel and Distributed Computing, 1994. T.R. Gruber. A translation approach to portable ontology specifications. Knowledge Acquisition, 5(2):199-220, 1993. J. Hendler, K. Stoffel, and A. Mulvehill. High Performance Support for Case-Based Planning Applications. Advanced Planning Technology. MIT/AAAI Press, Menlo Park, CA., USA, May 1996. J.F. Horty, R.H. Thomason, and D.S. Touretzky. A skeptical theroy of inheritance in nonmonotonic semantic networks. Technical Report CMU-CS-87-175, Carnegie Mellon, Department of Computer Science, Pittsburgh, PA, USA, Oct. 1987. B.P Kettler. Case-based Planning with a High-Performance Parallel Memory. PhD thesis, Dept. of Computer Science, University of Maryland, College Park, 1995. Brian B. Kettler. Case-based Planning with High-Performance Parallel Memory. PhD thesis, University of Maryland, College Park, November 1995. K. Knight and S. Luk. Building a large knowledge base for machine translation. In AAAI-94

[~].

15. Douglas B. Lenat and R.V. Guha. Building Large Knowledge-Based Systems. Addison-Wesley, 1991. 16. Robert M. MacGregor. A description classifier for the predicate calculus. In AAAI-94 [1]. 17. Joel H. Saltz, Ravi Mirchandaney, and Kay Crowley. Run-time parallelization and scheduling of loops. IEEE Transactions on Computers, pages 603-611, 1991. 18. K. Stoffel and J. Hendler. An efficient inferential-distance-ordering algorithm for parallel and distributed systems, submitted for publication, 1996. 19. K. Stoffel, J.A. Hendler, and J. Saltz. High performance support for very large knowledge bases. In Proc. Frontiers of Massively Parallel Computing. Feb. 1995. 20. Merwyn Taylor. Hybrid kb-db. In Proceedings of the 13th National Conference of the American Association for Artificial Intelligence. AAAI Press, MIT Press, 1996. to appear. 21. L. Watanabe and L. Rendell. Effective generalization of relational descriptions. In AAAI Eighth National Conference on Artificial Intelligence. 1990.

115 A. UM-Translog Queries ;;; QUERY 1: "find all plans for getting a package from a LOC to a LOC in DIFF city connected by a direct route"

(query! '(:and (#!instanceOf ?plan #!E-Plan) (#!instanceOf ?goal #!CE-Located-At) (#!everyInstanceOf ?pkg #!Package) (#!everyInstanceOf ?oloc #!Place) (#!everyInstanceOf ?dloc #!Place) (#!instanceOf ?ocity #)City) (#!instanceOf ?dcity #!City) (#!instanceOf ?sit #!Initial-Situation) (#!instanceOf ?sval #!Situated-Value) (#!instanceOf ?pkg-pvar #!Plan-Variable) (#!instanceOf ?dest-pvar #!Plan-Variable) (#!everylnstanceOf ?route #!Route) (#!plan-Goal= ?plan ?goal) (#!thing= ?goal ?pkg-pvar) (#!var-Value= ?pkg-pvar ?pkg) (#!location= ?goal ?dest-pvar) (#!var-Value= ?dest-pvar ?dloc) (#!situation-Before= ?plan ?sit) (#!location-sit= ?pkg ?sval) (#!situation= ?sval ?sit) (#!sslot-Val= ?sval ?oloc) (#!in-City ?oloc ?ocity) (#!in-City ?dloc ?dcity) (#!neq ?ocity ?dcity) (#!route-Origin= ?route ?ocity) (#!route-Destin= ?route ?dcity)

)) ;;; QUERY2: "find all plans (query! ' ( :and (# !instanceOf (# !instanceOf (#!everyInstanceOf (# !instanceOf (# !instanceOf (# !vehicle= (# !var-Value= (# !destin= (# !var-Value= (#!action-Of-Plan

using a particular train station, Hub-Ts" Tact ?plan ?train ?p-varp ?pvara Tact ?p-varp Tact ?pvara Tact

# !P-Move-Vehicle) # !E-Plan) #}Train) # !Plan-Variable) # !Plan-Variable) ?p-varp) ?train) ?pvara) # !Hub-Ts) ?plan)

)) ;;; QUERY 3: "find all top-level plans using a Regular truck (uses IDO and transitive inheritance)" (query ! ' ( :and (# !everyInstanceOf ?truck1 # !Regular-Truck) (# !instanceOf ?act # !P-Move-Vehicle) (# !instanceOf ?plan # !E-Plan) (#! instanceOf ?plant #!E-Plan)

116

(#!instanceOf (#!instanceOf (#!var-Value= (#!vehicle= (#!action-Of-Plan (#!tmember-Of (#!plan-of-Case ))

?case ?pvart ?pvart ?act ?act ?plan ?plant

#!Case) #!Plan-Variable) ?truckl) ?pvart) ?plan) ?plant) ?case)

;;; QUERY 4: "find all plans for getting an object from a LOC to a LOC in SAME city" (query! '(:and (#!instanceOf ?plan #!E-Plan) (#!instanceOf ?goal #!CE-Located-At) (#!everyInstanceOf ?pkg #!Package) (#leveryInstanceOf ?oloc #!Place) (#!everyInstanceOf ?dloc #!Place) (#!instanceOf ?city #!City) (#!instanceOf ?sit #!Initial-Situation) (#!instanceOf ?sval #!Situated-Value) (#!instanceOf ?pkg-pvar #!Plan-Variable) (#!instanceOf ?dest-pvar #!Plan-Variable) ?plan ?goal) (#!plan-Goal= ?goal ?pkg-pvar) (#!thing= (#!var-Value= ?pkg-pvar ?pkg) (#!location= ?goal ?dest-pvar) (#!var-Value= ?dest-pvar ?dloc) (#!situation-Before= ?plan ?sit) (#!location-sit= ?pkg ?sval) (#!situation= ?sval ?sit) (#!sslot-Val= ?sval ?oloc) (#!in-City ?oloc ?city) (#!in-City ?dloc ?city) )) ;;; QUERY 5: "find all plans DIFF city" (QUERY! '(:and (#!instanceOf (#!instanceOf (#!everyInstanceOf (#!everyInstanceOf (#!everyInstanceOf (#!instanceOf (#!instanceOf (#!instanceOf (#!instanceOf (#!instanceOf (#!instanceOf (#!plan-Goal= (#!thing= (#!var-Value= (#!location= (#!var-Value=

for getting an object from a LOC to a LOC in

?plan #!E-Plan) ?goal #!CE-Located-At) ?pkg #!Package) ?oloc #!Place) ?dloc #!Place) ?ocity #!City) ?dcity #!City) ?sit #!Initial-Situation) ?sval #!Situated-Value) ?pkg-pvar # !Plan-Variable) ?dest-pvar # !Plan-Variable) ?plan ?goal) ?goal ?pkg-pvar) ?pkg-pvar ?pkg) ?goal ?dest-pvar) ?dest-pvar ?dloc)

11"7 (#!situation-Before= ?plan?sit) (#!location-sit= ?pkg ?sval) (#!situation= ?sval ?sit) (#}sslot-Val= ?sval ?oloc) (#}in-City ?oloc ?ocity) (#!in-City ?dloc ?dcity) (#}neq ?ocity ?dcity) ))

;;; QUERY 6: "find compatible packages and vehicles" (query! '(:and (# !everyInstance0f ?pkg # !Package) (#!ancestor ?ptype #!Package) (#!ancestor ?vtype #!Vehicle) (#!everyInstance0f ?veh #}Vehicle) (#!instance0f ?pkg ?ptype) (#!instance0f ?veh ?vtype) (#!pv-Compatible= ?ptype ?vtype) ))

118 Dr. Kilian Stoffel

Dr. Kilian Stoffel is an affiliate research scientist at the University of Ma .ryland where he has worked in the Information Technology Laborato .ry since November 1994. His main research interests lie in Symbolic AI, Knowledge Representation and High Performance Computing. J a m e s A. H e n d l e r

Dr. Hendler is an associate professor and head of the Advanced Information Technology Laboratory at the University of Maryland, where he has worked since Janua .ry, 1986. He is the author of the book "Integrating Marker-Passing and Problem Solving: An activation spreading approach to improved choice in planning" and is the editor of'Expert Systems: The User Interface," "Readings in Planning" (with J. Allen and A. Tate), and "Massively Parallel AI" (with H. Kitano). Dr. Hendler was the recipient of a 1995 Fulbright Fellowship and has authored over 100 technical articles in the areas of artificial intelligence, high performance computing, and robotics. J o e l Saltz Associate Professor, Computer Science and head of High-Performance Systems Software Laboratory, University of Maryland. Joel Saltz leads a research group at the University of Maryland, College Park whose goal is to develop technology to make it easier for users to develop multiprocessor applications. Projects include the development of runtime support for parallel compilers, development of class libraries, optimizations for high performance I/O and development of techniques to link multiple separately parallelized applications.

Parallel Processing for Artificial Intelligence 3 J. Geller, H. Kitano and C.B. Suttner (Editors) 9 1997 Elsevier Science B.V. All rights reserved.

121

A Hybrid Approach to Improving the Performance of Parallel Search* Diane J. Cook Department of Computer Science and Engineering Box 19015 University of Texas at Arlington Arlington, TX 76019 (817) 273-3606 [email protected] Many artificial intelligence techniques and applications rely on performing heuristic search through large problem spaces. Iterative-Deepening-A* (IDA*) search has proven to be effective for large search spaces, because it requires no intermediate state storage and is guaranteed to find optimal solutions. However, the time taken to perform IDA* search on real-world tasks often prevents the everyday usage of AI techniques. Parallel processing can considerably reduce the time spent in search, and can thereby speed up AI applications. This paper describes HyPS, a hybrid parallel window / distributed tree search algorithm. Using this algorithm, the set of available processors is divided into clusters. Each cluster searches simultaneously through the same search space, but to a unique cost threshold. Within each cluster, the search space is divided so that an individual processor will search a fraction of the total search space. Operator ordering and load balancing techniques are used to further improve the performance of HyPS. Results on two real-world and one artificial domain show a substantial performance improvement over serial search algorithms, and indicate an improvement over existing parallel search approaches. In this paper we also describe a mechanism for automatically selecting the optimal number of clusters to use. 1. I N T R O D U C T I O N Heuristic search provides the driving force for many applications of artificial intelligence including problem solving, robot motion planning, concept learning, theorem proving and natural language understanding [9,21]. Computational complexity is a major limitation of search, thus the research community is continually t .rying to develop more efficient search algorithms. Parallel search algorithms significantly increase the size of the search space that can be traversed in a given amount of time. Parallel search techniques have been implemented on MIMD [3,20,21] and SIMD [4,7,8,12,19] architectures. Because of the overwhelming size of real-world AI applications, and because of the increasing power and accessibility *This work is supported by National Science Foundation grants IRI-9308308 and IR1-9502260.

122 of parallel computers, there exists a constant need for improvement of parallel search algorithms. This paper introduces a hybrid parallel search technique (HyPS) that improves the performance of search using a MIMD architecture. The idea behind the approach is to blend the strengths of existing parallel search algorithms. In particular, HyPS merges the power of a parallel window search with that of a distributed tree search. The resulting algorithm offers improvements over either approach used by itself. The addition of load balancing and operator ordering further improves the performance of the HyPS system. The remainder of this paper describes the HyPS system and demonstrates its performance on two well-known AI application areas as well as an artificially-generated search space. Section 2 defines the search problem and describes the basic search techniques employed by HyPS, and section 3 provides an overview of existing parallel search techniques. The following section introduces the HyPS approach along with operator ordering and load balancing extensions. Section 5 demonstrates the performance of HyPS applied to two real-world problems and one artificial search-intensive problem. Section 6 overviews a technique for automatically selecting the number of clusters to use for a given problem. We conclude this paper with observations and a discussion of future research directions.

2. H E U R I S T I C

SEARCH

Heuristic search techniques are used to find a sequence of operators that lead from the initial state (root node) of a problem to a goal state (goal node). An example search tree is given in Figure 1. Search begins by evaluating the root node and generating the children of the root node. At each later step, one of the previously generated child nodes is evaluated and its children are generated. This process continues until a node is evaluated that meets the goal criteria. Search time can be reduced by using estimated distances to the goal (heuristics) to direct the search. A* is a type of heuristic search algorithm which selects a node from the search queue that minimizes the function f(n) - g(n) + h(n). In this function, g(n) represents the cost of the path from the initial state to node n, and h(n) represents the heuristic estimate of the least-cost path from node n to a goal node. If the function h(n) never overestimates the distance to the goal, A* is guaranteed to find an optimal (least-cost) solution. The main drawback of any heuristic search algorithm is that the memory requirement is exponential in the depth of the tree [17]. The resulting demand for memo .ry often exceeds the resources of available machines. Iterative-Deepening-A* (IDA*) search [10] provides a solution to this problem. IDA* performs a series of incrementally-deepening depth-first searches through the search space. In each iteration through the space, the depth of the search is controlled by the cost threshold. The cost of a node is calculated using the A* function f(n) = g(n)+ h(n). The initial cost threshold is computed as the estimated distance from the root node to the goal. Search down any branch of the space terminates when the f(n) value of a generated child node exceeds the cost threshold for that iteration. If a goal node is not found during a given iteration, the search starts back at the root node, but the cost threshold is set to the minimum f(n) value in the search space that exceeded the previous threshold. Successive iterations continue until a goal node is found.

123 INITIAL STATE

I

GOAL STATE

Figure 1. Application search tree

IDA* also guarantees optimal solutions if h(n) does not overestimate the distance to a goal. Although redundant work is performed, the number of nodes that are expanded more than once is a small fraction of the total work. Unlike A* search, the memory requirement for IDA* is linear in the depth of the search space. 3. P A R A L L E L

SEARCH

A number of researchers have explored methods for improving the efficiency of search. These methods include making use of background knowledge, learning search macro operators, and designing parallel search algorithms. Parallel architectures have been shown to be useful in reducing the amount of serial time spent in search by dividing the work among multiple processors. Many of the existing parallel search efforts can be classified as parallel window searches or distributed tree searches. 3.1. P a r a l l e l W i n d o w S e a r c h One popular type of parallel search algorithm is a parallel window search (PWS), introduced by Powley and Korf [20]. Using PWS, each processor is given a copy of the entire search tree and a unique window size or cost threshold. The processors search the same tree to different thresholds simultaneously. If a processor completes an iteration without finding a solution, it is given a new unique threshold (deeper than any threshold yet searched) and begins a new search pass with the new threshold. The first processor to reach a solution informs the rest of the processors. The remaining processors stop their search if this solution is acceptable. If an optimal solution is desired, processors searching at lower thresholds finish their current iteration and the least-cost goal is returned. Given the search tree shown in Figure 1, the division of work using PWS is illustrated in Figure 2. One advantage of parallel window search is that the redundant search inherent in IDA* is not performed serially. A serial algorithm would search the space to threshold t, t h e n

124 Processor 1 I N I T I ~ ~

Processor 2 INITIAL ....

Processor 3 INITIAL ---

~~

O. O 4 O

........GOAL threshold I : h(root)

threshold 2 = thresholdI + i

threshold 3

=

threshold I + 2i

Figure 2. Division of work in parallel window search

would search the same space to threshold t plus some increment (t + i), then to threshold t + 2i, etc., until a goal node is found. On each iteration, all of the nodes expanded in the previous iteration are expanded again. Using multiple processors, this redundant work is performed concurrently. A second advantage of parallel window search is the improved time in finding a first solution. If the solution density is high (there are many solutions in the search space), the depth-first search used by IDA* may find a deep solution much more quickly than an optimal solution. Parallel window search can take advantage of this type of search space. Processors that are searching beyond the optimal threshold may find a solution down the first branch they explore, and can return that solution long before other processors finish their search iteration. This will often result in superlinear speedup in comparison to the serial algorithm, because the serial algorithm will always increment the cost threshold by the least possible amount and not look beyond the current threshold. On the other hand, parallel window search can face a decline in efficiency when the number of processors is significantly greater than the number of iterations required to find an optimal (or a first) solution. This situation will occur when a machine is used that offers many processors, yet few iterations are required because the heuristic estimator is fairly accurate. 3.2. D i s t r i b u t e d Tree Search An alternative approach to parallel search distributes the search space among available processors and is referred to here as distributed tree search (DTS) [11,21]. Using our version of DTS, one processor traverses the search tree until there are at least as many nodes in the queue as there are available processors. Once a sufficient number of nodes have been generated, the first processor passes a unique node from the search queue to the remaining processors. Each processor is thus responsible for the entire subtree rooted at the node it received. The processors perform IDA* on their unique subtrees simultaneously. All processors search to the same threshold. After all processors have

125

~llllliii,liiiiiil|liil|lliiiiliilii,lllliiliiiiiiliii|llil~ i 9 = expanded on all three iterations ~ : | i ~ = expanded on last two iterations i 9 i 0 = expanded on last iteration | I,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,~

i ! GOAL

Figure 3. Division of work in distributed tree search

finished a single iteration, they begin a new search pass through the same set of subtrees using a larger threshold. Given the search tree shown in Figure 1, processors running DTS search the spaces shown in Figure 3. One advantage of DTS is that no processor is performing wasted work beyond the goal depth. As the algorithm searches the space completely to one threshold before starting the search to a new threshold, none of the processors is ever searching at a level beyond the level of the optimal solution. It is possible, however, for DTS to perform wasted work at the goal depth. For example, in Figure 3 processor 3 searches nodes at the goal level that would not be searched in a serial search algorithm moving left-to-right through the tree. A disadvantage of DTS is the fact that processors are often idle. This is because a processor that finishes an iteration quickly must wait for all other processors to finish before starting the next iteration (in order to ensure optimality). This idle time can make the system ve.ry inefficient and reduce the performance of the search application. The efficiency of this approach can be improved by performing load balancing between neighboring processors working on the same iteration. Load balancing itself will generate overhead and communication costs, however. A second disadvantage of DTS is that it does not provide a mechanism for finding a quick first solution. The original DTS algorithm described by Kumar and Rao assigns the root node of the space to the first processor, and load balancing must be performed for other processors to receive initial work. We modified this algorithm to reduce load balancing time by distributing nodes to all processors from the host.

3.3. Other Parallel Search Techniques Both parallel window search and distributed tree search are implemented on MIMD architectures. Additional MIMD and SIMD approaches have been suggested which have

126 not yet been incorporated into the HyPS system. Bidirectional search has been used to search the space forward from the initial state and backward from the goal state concurrently [13]. Variations on the distributed tree search technique have been tailored to specific applications such as robot path planning [2]. Yet another body of research focuses on concurrent random movements through a search space that cover the search space quickly [6]. Ferguson, Powley and Korf describe a SIMD search technique in which, like DTS, the work is distributed among processors and each of the processors search to the same threshold [18,19]. The distribution relies on a series of copies which takes O(log p) steps, where p represents the number of available processors. Another SIMD search techniques which relies on fast information distribution and a distributed tree search is the Massively Parallel IDA* (MIDA*) system [4]. MIDA* also takes O(log p) steps to perform distribution, but search is performed as the information is distributed, so that the amount of work performed during the parallel IDA* step is reduced. A third SIMD search techniques again uses distributed tree search, but focuses on techniques for load balancing between processors during the parallel IDA* stage [12]. 4. H y P S We introduce a new type of parallel search that combines characteristics of existing approaches. We refer to the algorithm as Hybrid Parallel Search (HyPS) [14-16]. In particular, HyPS combines parallel window search with distributed tree search. As the experiments will show, the combination of search techniques outperforms either technique used in ~isolation. HyPS divides t h e set of processors into groups called clusters. Each cluster performs parallel window search - - each cluster receives a copy of the entire search space, and each cluster searches that Space to a unique cost threshold. Within each cluster, the space is distributed among the cluster's processors and the processors perform distributed tree search. In this way, clusters perform parallel window search, and the processors within each cluster perform distributed tree search. HyPS has been implemented in C on a nCUBE and in C* on a Connection Machine 5. The next two subsections describe the distribution and search phases of the HyPS algorithm. The following subsections describe improvements that can be made using operator ordering and. load balancing.

4.1.' DistributiOn Phase During the distribution phase, the host program distributes work among the node processors. The host program first determines Jhow manyclusters will be used and divides the set of processors equally among the cluSters. The search tree is then expanded t o t h e point that distinct subtrees can be distributed to each processor within a cluster. The number of distinct subtrees being explored is equal to the number of processors avaiiable Within a cluster. For example, if we have n = 6 available processors and we wish to use two separate parallel windows, then HyPS will form two clusters, each with n/2 - 3 processors. Hence, the host program will have to divide the initial tree into three distinct subtrees, which in turn are distributed to the three processors in each cluster (see Figure 4). ~ ~

127

Cluster 2

Cluster 1 IN

INITIAL

.......

9

9

O

rocessor

1

Processor

Q

Processor

2

Processor

5

O

Processor

3

Processor

6

O

o

o

4

Figure 4. Space searched by two clusters, each with 3 processors

The host processor stores generated nodes in a queue (Q). The number of generated nodes is dependent upon the number of distinct subtrees required for the distributed tree search. The steps taken in the formation of the queue are detailed below. 1. Initialize Q to contain the root of the search tree. 2. While length(Q) < number of processors in each cluster do 9 Remove node n from head of Q. 9 Evaluate node n. If n is not a goal node, then generate the children of n and append them onto Q. 3. If length(Q) > number of processors in each cluster, compress nodes at end of Q. Note that using a level-by-level search, more nodes may be generated than available processors. In this case, nodes at the end of the search queue are removed from the queue and replaced by their parent nodes. This compression is performed as many times as needed until the length of the queue is less than or equal to the number of processors in each cluster. If the final queue length is less than the number of processors needing information, child nodes from the most recent compression are distributed to the idle processors. In this case, both the parent node and some of its children are being searched by separate processors, yielding some redundancy in the search process. We provide an example of the distribution algorithm for an instance of the Fifteen Puzzle problem where each cluster contains p = 6 processors. The expansion of the search tree is shown in Figure 5. In this picture, invalid nodes (the blank square cannot move beyond the board boundaries and cycles of length two are disallowed) are not shown. The final queue will contain the leaf nodes of this partial tree, and will be distributed

128

INITIAL STATE

, 812 723~

, 8112 I , ~ 21 13

lo 11 13 1~

14

1

9i

i

14

7

2

3

51

7

101~ 1, I1~

1 "9

6

2

5

3

, ~12 ~2 31~

io 11 13

l o 1111311~

14 1 ~ ~] [1! ;:9 6 7

2n

31

4

8 12

5

7

2

3

lo11~3~

immn ilUiNi ININI ImEEE

Figure 5. Distribution of work for a fifteen puzzle problem

to the respective processors in each cluster. Once distribution is complete, the signal is given to begin the search phase. 4.2.

Search

Phase

The search phase is performed in parallel by the processors within each cluster. To begin the search phase, the host sends the nodes defining the subtrees to the appropriate processors within each cluster. The cluster also receives the cost threshold to which it will search. Each processor then begins searching at the root of its subtree. The first cluster receives a cost threshold equal to the heuristic estimate of the distance from the initial state to the goal. All of the remaining clusters are given incrementally larger thresholds. For many problems, the appropriate increment can be predetermined in a way that guarantees no loss of optimality. Figure 4 illustrates the case where n = 6 processors are available and we wish to use two separate parallel windows. By changing the cluster size, a greater or lesser number of windows can be employed. When the number of clusters c is equal to the number of processors available on the machine, HyPS simulates pure parallel window search. When c is equal to 1, HyPS simulates pure distributed tree search. Because clusters search the space to unique depths and processors within each cluster search a unique portion of the space, it is unknown where the first solution found will lie, and whether its cost will be optimal or nonoptimal. HyPS can be used to specifically return an optimal solution or to return the first (optimal or nonoptimal) solution found. 4.3.

Operator

Ordering

Problem solutions can exist anywhere in the search space. Using IDA* search, the children are expanded in a depth-first search from left to right, bounded in depth by the cost threshold. If the solution lies on the right side of the tree, a far greater number of nodes must be expanded than if the solution lies on the left side of the tree. The left-to-right direction of search represents the order in which search operators are

129 Orlglnal Ordering: New Ordering:

0123

1320

0.." i

W Most promising node

Figure 6. Operator ordering example

applied. If information can be found to re-order the operators in the tree from one search iteration to the next, the performance of IDA* can be greatly improved. As an example, one of the tested Fifteen Puzzle problem instances can require expansion of 900,587 search nodes for the best ordering of operators, yet requires expansion of 4,570,051 search nodes for the worst operator ordering. HyPS attempts to find a good operator ordering for each iteration based on information gained from the previous iteration [3]. The reduction in number of expanded nodes can be exponential for some problem instances. HyPS stores the order of operator applications that lead to the most promising node at the bottom of the subtree explored in the previous search pass. The "most promising" node is the one that has the minimum h value. Since all nodes at the bottom of a subtree represent equal f values (by definition of IDA*), nodes with smaller h values are expected to be more accurate. Consider the situation in which a car needs to be fixed. A customer generally feels more confidence about the statement that his car will be ready in 2 days (h = 2) if it has already been in the shop 28 days (g = 28), than he would feel about the statement that his car will be ready in 28 days (h = 28) if the car has only sat in the shop for 2 days (g = 2). If, as in many cases, the error in the heuristic function h grows monotonically with the distance from the goal, nodes with the lowest h values represent the most accurate estimated distances to the goal and contain fewer nodes in their subtrees [20]. The following example illustrates the operator ordering technique. Figure 6 shows the search tree expanded on one iteration of IDA*. The most promising leaf node is starred.

130 illllll

9-: _-

IIIIIIIIIIIIIIIIIIIIII

IIIIIIIIIIIIIII

II

IIIIIII

IIIIi

Clusterl

:

"

......

. ........................

illl

ll

llllllllllllllllllll

Illllllllll

ll

lllllllllllll

llllll

:_.

:"9 C l u s t e r 2

: ,

:

i

i

J

-: . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

, .......................

J

Figure 7. Load balancing of two clusters, each with 4 processors

The current operator order is 0123 (always try operator 0 first, operator 1 next, operator 2 next, and operator 3 last), but the sequence of operators that leads to the most promising node at the current cutoff point is 133202, so the operator order is changed to reflect this search direction. Because operator 1 appears first in the path to the most promising node, operator 1 will always be tried first. The next unique operator in the sequence is operator 3, then operator 2, and finally operator 0. This ordering is updated at the end of each parallel window search iteration. 4.4.

Load

Balancing

Within

a Cluster

In the same way that PWS is aided by the addition of operator ordering, DTS benefits from the addition of load balancing. Distributed tree search suffers greatly due to processor idling. Load balancing can be added to improve the intra-cluster performance by reducing this idle time. Using load balancing, a processor that has finished its current search iteration asks for work from its neighbor and helps that neighbor to complete its search iteration. We have added load balancing to HyPS to improve the overall performance. Load balancing here is implemented using a ring model of communication. Consider the case where we have eight available processors divided into two clusters. Load balancing will be performed within each of these four-processor clusters. If processor Pl finishes work, then Pl asks for work from its right neighbor, P2. Each processor performs a similar type of request except for the last processor in the ring, such as p4, which requests work from Pl. Figure 7 depicts this communication model. When work is requested from a processor, the processor first determines if it has work to spare (the number of nodes on the stack is greater than a specified threshold). If there is work to spare, the donating processor sends information corresponding to the node at the bottom of the stack, or the node closest to the root which has children left to expand. If the processor has no work to spare, then the requesting processor must remain idle until the search is finished or a new threshold is specified. In this load balancing algorithm, communication is restricted to the adjacent processors within the cluster. Load balancing was added to HyPS to see how the performance of DTS and PWS would be affected. To improve the performance further, more advanced load balancing techniques could be employed [1,23]. As the experiments demonstrate, integrating the

131

1

2

3

4

5

6

7

8

9

i0

ii

12

13

14

15

Figure 8. The fifteen puzzle

additional strategies of load balancing and operator ordering improve the performance of HyPS. 5. E X P E R I M E N T S This section demonstrates the performance of HyPS on two real-world applications and one artificially-crafted domain. The experiments on real-world domains reveal that 1) HyPS does provide a substantial speedup over a serial algorithm, 2) operator ordering and load balancing can in many cases improve the performance of HyPS, and 3) the optimal number of clusters varies, verifying that PWS and DTS perform better in combination than in isolation. We will first introduce the two real-world applications and provide the experimental results, then describe our artificial domain experiments. 5.1. T h e F i f t e e n P u z z l e One well-known discrete optimization problem (DOP) is the Fifteen Puzzle. It consists of a 4 x 4 grid containing tiles numbered one to fifteen, and a single empty tile position called the blank tile. A tile can be moved into the blank position from an adjacent position, resulting in the four operators up, down, left, and right. Given the initial and goal configurations, the problem is to find a sequence of moves to reach the goal. An example of a goal configuration is given in Figure 8. In our problem instances, the heuristic estimate function used is the Manhattan Distance Function, which is known to be an admissible heuristic for the Fifteen Puzzle problem. The Fifteen Puzzle problem can generate approximately 240 unique problem states [7], thus this represents a large search problem which can benefit from an efficient parallel algorithm. 5.2. R o b o t A r m M o t i o n Planning Discrete optimization problems belong to the class of NP-hard problems. Parallel processing of these problems cannot reduce their worst-case run time to a polynomial without using an exponentially large number of processors. However, the average-time complexity of heuristic search algorithms for many of these problems is polynomial. Robot motion planning, speech understanding, and task scheduling are some of the DOPs which require real-time solutions. Parallel processing of these problems can bring their real-time

132

Figure 9. Robot arm motion planning problem instance

solutions closer to being realized. Traditional motion planning methods are ve.ry costly when applied to a robot arm, because each joint has an infinite number of angles to which it can move from a given configuration, and because collision checking must be performed for each arm segment. Craig provides a detailed description of the calculations necessa .ry to determine the position of the end effector in the 3D workspace given the current joint angles [5]. For our experiments, we use the parameters defined for the Puma 560 robot arm with six degrees of freedom shown in Figure 9. The size and layout of the room is the same for each of our test problems.

5.3. Results from real-world experiments We tested the performance of HyPS for the Fifteen Puzzle and Robot Arm Motion Planning domains using the basic HyPS approach as well as HyPS with operator ordering and load balancing added. Each of these experiments were run on 64 processors of an nCUBE 2. Table 1 lists the speedup that resulted from each selected number of clusters, averaged over 20 instances of the Fifteen Puzzle problem. Speedup over the corresponding singleprocessor algorithm was generated using basic HyPS, HyPS with operator ordering, HyPS with load balancing, and HyPS with both operating ordering and load balancing. Results were collected for finding a first solution and an optimal solution. In all of the result summaries, the speedup of the best cluster distribution is highlighted. As table 1 indicates, all versions of the program realize significant speedup over the serial algorithm. Speedup is greater when the program stops with a first solution, because

133 Table 1 Speedup for Fifteen Puzzle domain Number of clusters Algorithm Basic Operator Ordering Load Balancing Operator Ordering / Load Balancing

1 2 4 (DTS) ~2.4 ~6.6 24.2 12.4 2 2 . 0 45.9 18.4 43.1 43.1 18.4 4 4 . 2 82.3 17.6 28.2 2 4 . 9 17.6 3 0 . 6 47.1 41.3 2 3 . 1 4 0 . 7 41.3 2 4 . 3 7 9 . 2

8

16

32

64

Avg

Solution

12.2 28.9 19.5 50.9 15.3 33.2 19.9 52.1

Optimal First Optimal First Optimal First Optimal First

(pws) ~6.8 46.4 16.2 85.3 17.3 57.3 19.6 74.8

6.~ 33.4 6.2 53.5 9.2 36.7 6.5 87.3

6.5 20.3 5.4 43.4 7.5 26.8 5.5 41.1

3.0 22.2 3.9 29.5 2.7 16.3 2.7 16.5

the serial algorithm is constrained to always find an optimal solution. In addition, both operator ordering and load balancing improve the speedup of HyPS, and the best speedup is realized when all of these techniques are used. Superlinear speedup may occur when finding a first solution because the comparison is made to a serial algorithm finding an optimal solution. Superlinear speedup may also result when the goal node lies on the right side of the space, because the space is divided evenly among processors and an individual processor will find that solution before a serial algorithm that must search the entire left side of the space before reaching the goal. Table 2 lists the number of problems for which the corresponding number of processors yields the best performance. Note that in no case does pure DTS (number of clusters - 1) or pure PWS (number of clusters - 64) consistently perform best. In general, performance improves with a greater number of clusters when looking for a first solution. This occurs often in domains like the Fifteen Puzzle where the solution density (percentage of nodes representing goals) is high, and increasing the number of windows increases the chance of quickly finding a goal. As expected, operator ordering and load balancing affect the ideal number of clusters. Because operator ordering benefits PWS, the optimal number of clusters increases for these tests. When load balancing is used in isolation, the performance of DTS improves and the optimal number of clusters becomes lower. The greatest number of clusters and the greatest speedup is realized when both operator ordering and load balancing are used to find a first solution. The results for the Robot Arm Motion Planning domain are listed in table 3. The results are averaged over eight problem instances in which only optimal solutions are sought. As before, speedup is obtained over the serial algorithm and performance improves when operator ordering and load balancing are added. However, this domain differs from the Fifteen Puzzle domain in two main features. First, many of the solutions in this domain lie on the far right side of the tree. For this reason, superlinear speedup is often realized when

134

Table 2 Optimal number of clusters for Fifteen Puzzle domain

Number of clusters Algorithm

1

2

4

8

16

32

64

Avg

Solution

Basic

4 1 1 0 5 2 3 0

7 3 6 3 7 3 6 1

3 6 7 7 3 5 6 8

6 6 3 5 4 8 3 6

0 3 1 3 0 1 1 3

0 0 2 1 1 0 1 1

0 1 0 1 0 1 0 1

3.9 10.1 7.3 10.9 3.2 8.6 5.6 11.3

Optimal First Optimal First Optimal First Optimal First

Operator Ordering Load Balancing Operator Ordering / Load Balancing

Table 3 Speedup for Robot Arm Motion Planning domain

Number of clusters Algorithm Basic Operator Ordering Load Balancing Operator Ordering / Load Balancing

1 2 (DTS) 14.7 21.4 13.1 38.7 12.9 21.2

4

8

16

32 2.1 2.0 2.3

64 (PWS) 2.1 2.0 2.1

70.2 230.6 78.0

27.1 891.7 27.0

10.6 1671.1 14.1

12.8 84.9

155.0

1518.4

354.1

Avg 18.5 356.2 19.7

2.8

1.8

266.2

135 Table 4 Optimal number of clusters for Robot Arm Motion Planning domain

Number of clusters Algorithm

1

2

4

8

16

32

64

Avg

Basic Operator Ordering Load Balancing Operator Ordering / Load Balancing

0 0 0

0 0 0

7 0 7

1 7 1

0 1 0

0 0 0

0 0 0

4.5 9.0 4.5

0

0

1

6

1

0

0

8.5

operator ordering is employed. Second, the branching factor is nearly constant throughout the tree, so the tree is very uniform. As a result, the load is balanced and addition of load balancing does not greatly improve performance. In fact, in some cases load balancing can actually degrade performance due to the time taken to query neighboring processors for available work. Table 4 lists the number of problems for which a given number of clusters performed best. As before, operator ordering causes an increase in the ideal number of processors. Because load balancing did not significantly change performance, the ideal number of clusters remains the same when load balancing is added. For both of these problems, the optimal number of clusters was not constant, but varied with characteristics of the problem domain and with the inclusion of operator ordering and load balancing. 5.4. A r t i f i c i a l D o m a i n Intuitively, selecting the number of clusters to use for a given search application seems to be affected by factors such as the branching factor of the tree, the load imbalance, the accuracy of the heuristic function, and the position of the solution in the tree. To isolate the factors that affect the performance of parallel search, we constructed an artificial search space. In this section, we will describe the artificial search space and show results of experiments using this space. The artificial search space is represented as a tree. Instead of storing the entire space in memo~., portions of the space are generated as needed while the algorithm traverses the tree. The following characteristics of the tree can be adjusted by the user. 9 Maximum branching factor. The user can select a maximum branching factor. No node in the tree will have more children than this specified value. 9 Load imbalance. A real number between 0 (perfectly balanced) and 1 (perfectly imbalanced) is used to indicate the amount of imbalance in the tree, and thus in the distributed work. An imbalanced tree contains the greatest number of nodes in the middle of the tree, and the left and right sides contain sparse subtrees. 9 Accuracy of heuristic estimate. A real number between 0 and 1 is used to indicate

136

\ \ m

o~

O 1 2

3 Branching

4

5

Factor

Figure 10. Artificial Experiment #1

the accuracy of the heuristic estimator. Because h should never overestimate the true distance to the goal, h is defined as e r r o r h 9 d i s t a n c e . 9 Cost of optimal solution. Each move is assigned a cost of 1, so the optimal cost is the depth in the tree at which the optimal solution can be found. 9 Position of optimal solution. A sequence of moves is supplied to define the path to the first optimal goal found in a left-to-right search of the tree. 9 Solution density. A real number between 0 and 1 represents the probability that any given node found after the first optimal solution is also a goal node. Using this artificial domain, we can test several hypotheses about the features of a search space that affect the optimal number of clusters to be used. Note that there are no random distributions used in the generation of the tree, because the tree must maintain the same structure for each iteration of IDA*. Instead, eve .ry feature of a node is determined as a function of the node's position in the search tree. All of the experiments described in this section were conducted using 32 processors on an nCUBE 2. The first experiment compares the optimal number of clusters to the average branching factor in the tree. When the average branching factor is low, there is little work to distribute and better performance is obtained by parallelizing iterations of IDA* using PWS. As the branching factor increases, there is more work to distribute and a lower number of clusters is favored. To verify this hypothesis, we ran HyPS on four different artificially-generated search spaces with the average branching factor ranging from 2 to 5. In each case, the tree

137

/

30

r~ S.,

_= u

/

...i

/

0

0.0

0.4

0.8

Imbalance

Figure 11. Artificial Experiment # 2

is perfectly balanced, the heuristic estimate is 0.8 (fairly accurate), the optimal cost is 16, and the first optimal solution is positioned on the far right of the tree. No operator ordering or load balancing is employed. As Figure 10 shows, the hypothesis is verified by our experiment. When the branching factor is at the low end, the ideal number of clusters is 8. As the branching factor increases, the optimal number of clusters decreases eventually down to 1. In the second experiment, we verify that a nonuniform tree requires a greater number of clusters. Figure 11 shows the result of this experiment. We ran HyPS on six trees with load imbalance ranging from 0 to 1. For each tree, the solution is positioned in the middle of the tree, the branching factor is 3, the optimal cost is 15, and no load balancing or operator ordering is used. As the imbalance increases, more windows are required. This is due to the fact that with a high level of imbalance, many of the subtrees are very small and the processors searching these trees will be idle. As the first experiment in this section demonstrated, a high branching factor results in a lower optimal number of clusters. In contrast, a high number of IDA* iterations and a solution positioned on the left side of the tree will result in a higher optimal number of clusters. The third experiment controls the number of IDA* iterations required by varying the accuracy of the heuristic estimator function and by varying the position of the first optimal solution. For each run, the branching factor is 3 and the optimal cost is 15. While the accuracy of h varies from 0.2 to 1.0, the position of the first optimal solution also varies from the far left side of the tree to the far right side of the tree. Figure 12 shows that in fact with a low accuracy and solution on the left, more clusters are needed than when h is accurate and the first optimal solution appears on the right side of the

138

30

20

~= lO

0.2

0.6 Accuracy of h

1.O

Figure 12. Artificial Experiment #3

tree. The last experiment focuses on the effects of employing operator ordering. In experiment four, we demonstrate that operator ordering becomes more effective the farther to the right the solution lies. Each search tree for experiment four has a branching factor of 4, a heuristic estimate of 0.2, a uniform tree, and an optimal cost of 12. The solution is first position 0/5 of the way through the tree (far left), and moved successively further to the right (positions of 1/5, 2/5, 3/5, 4/5, and 5/5 through the tree). For each test case, HyPS is run with and without operator ordering. As Figure 13 verifies, operator ordering causes a greater increase in the ideal number of clusters and a more significant speedup as the solution is positioned farther to the right in the tree. The figure on the left graphs the increase in optimal number of clusters with the addition of operator ordering, and the figure on the right graphs the speedup that results from employing operator ordering. In this section we have used an artificial domain to identify features that contribute to optimizing the performance of HyPS. In the next section, we will describe an architecture that uses these features and a set of rules to automatically select the number of clusters. 6. A U T O M A T I C A L L Y

SELECTING OPTIMAL NUMBER

OF CLUSTERS

In this section we describe our method of automatically selecting the number of clusters that will optimize the performance of HyPS. To make this decision, we note that many characteristics of a search space can be determined by looking at the first few levels of a tree. In particular, by exploring just a few levels down in the tree we can estimate factors such as the average branching factor, the average error of the heuristic estimate function, the possible location of the optimal solution, and the amount of load imbalance.

139

/

|

I _/

/L J

/ / 0.0

Solution P o s i t i o n

0.4

0.8

Solution P o s i t i o n

Figure 13. Artificial Experiment # 4

Our architecture makes use of the host distribution mechanism to collect statistics about the search space. The amount of time required to search the first few levels of the tree is minimal, and valuable information can be gleaned about the nature of the problem. Using results from the real-world and artificial domains, rules were constructed to select the ideal number of clusters. Statistics are gathered about the problem space as the host searches a few levels down in the tree. The statistics are used to calculate the number of clusters for HyPS to use for the rest of the search. This method of gathering information about the search space to make decisions about the parallel search is similar to Suttner's SPS model, in which a small portion of the space is searched serially to determine whether the problem is a good candidate for parallelization, to control redundancy, and to determine an effective distribution scheme [22]. Using this architecture, we reran the artificial domain experiments and compared results to those shown in section 5.4. In particular, Figures 14 through 17 graph the results of experiments 1 through 4 where the number of clusters is selected automatically by the system. In each case, the trend in selected number of clusters is the same as the trend for the verified optimal number of clusters, though actual numbers vary in a few cases. We also reran the two real-world application domains, using this architecture to select the number of clusters for HyPS. We compared the number of clusters pick by our architecture to the optimal number of clusters verified by the experiments in section 5. For the Fifteen Puzzle, the average error in optimal number of cluster selection averaged over all 20 problem instances for the basic HyPS system was 1.1. This represents one selection away from optimal: choosing to use 2 clusters instead of 1, or 4 clusters instead of 8. For these results, the host system searched six levels into the tree. Similarly, the host searched six levels into the search tree for the Robot Arm Motion Planning domain. Averaged over the six problem instances using basic HyPS, the error in optimal number of clusters selection using our architecture was 0.125.

140

8-

16-

\

I..

I=i

\

6-

\

4-

2-

\

"~10_

_ 2

\ 3

J /

-

~14_

4

~

8 _-

5

0.0

Branching Factor

/

f 0.4

0.8

Imbalance

Figure 14. Artificial Experiment #1

Figure 15. Artificial Experiment #2

16-,

1.0k

A

/

ffJ

"•

0.812-

~

-

'~

0.6-

~ ~

~

8-

\

4 0

.~

\

Herror

Figure 16. Artificial Experiment #3

0.4_ -

~11 0 , 2 -

g r~ o.o-

/ !

/ ! !

Solution Position

Figure 17. Artificial Experiment #4

141 7. C O N C L U S I O N S A N D F U T U R E W O R K The goal of this research is to improve search-intensive applications by improving the performance of parallel search algorithms. This paper introduces the Hybrid Parallel Search algorithm HyPS, which improves the performance of existing parallel search techniques such as parallel window search and distributed tree search, by combining the benefits of both approaches. The system combines the two techniques by dividing the set of processors into clusters. The clusters perform parallel window search, while the processors within a cluster perform distributed tree search. The performance of HyPS is improved by the addition of operator ordering (which strengthens inter-cluster performance) and load balancing (which strengthens intra-cluster performance). The results of experiments applied to two problem domains reveal that HyPS outperforms both serial search algorithms and either PWS or DTS used in isolation. An artificial domain was used to identify the features that contribute to the selection of the optimal number of clusters. Finally, we introduced a mechanism for automatically selecting the number of clusters for each search domain that performed well in all three application domains. While the HyPS system has been shown to be effective in these three domains, we are continuing to apply the system to a variety of applications. As more data is gathered, we can use machine learning techniques to generate a set of parameter-setting rules for HyPS that achieves the greatest performance possible for parallel heuristic search.

REFERENCES

1. I. Ahmad and A. Ghafoor. Semi-distributed load balancing for massively parallel multicomputer systems. IEEE Transactions on Software Engineering, 17(10):9871004, 1991. 2. D. Challou, M. Gini, and V. Kumar. Parallel search algorithms for robot motion planning. In Proceedings of the AAAI Symposium on Innovative Applications of Massive Parallelism, pages 40-47, 1993. 3. D . J . Cook, L. Hall, and W. Thomas. Parallel search using transformation-ordering iterative-deepening a*. The International Journal of Intelligent Systems, 8(8), 1993. 4. D. J. Cook and G. Lyons. Massively parallel IDA* search. Journal of Artificial Intelligence Tools, 2(2):163-180, 1993. 5. J.J. Craig. Introduction to Robotics. Addison-Wesley Publishing Company, Inc, 1989. 6. W. Ertel. Massively parallel search with random competition. In Proceedings of the AAAI Symposium on Innovative Applications of Massive Parallelism, pages 62-69, 1991. 7. M. Evett, J. Hendler, A. Mahanti, and D. Nau. PRA*: Massively parallel heuristic search. Journal of Parallel and Distributed Computing, 25:133-143, 1995. 8. G. Karypis and V. Kumar. Unstructured tree search on simd parallel computers. In Supercomputing 92, pages 453-462, 1992. 9. H. Kitano. Massively parallel ai and its application to natural language processing. In Proceedings of the First International Workshop on Parallel Processing for AI, pages 99-105, 1991.

142 10. R. E. Korf. Depth-first iterative deepening: An optimal admissible tree search. Artificial Intelligence, 27:97-109, 1985. 11. V. Kumar and V. N. Rao. Scalable parallel formulations of depth-first search. In Kumar, Kanal, and Gopalakrishan, editors, Parallel Algorithms for Machine Intelligence and Vision, pages 1-41. Springer-Verlag, 1990. 12. A. Mahanti and C. Daniels. Simd parallel heuristic search. Artificial Intelligence, 60(2):243-281, 1993. 13. P. C. Nelson and A. A. Toptsis. Superlinear speedup using bidirectional and islands. In Proceedings of the 12th International Joint Conference on Artificial Intelligence, pages 129-134, 1991. 14. S. Nerur and D. J. Cook. Maximizing the speedup of parallel search using hyps. In

Proceedings of the Third International Workshop on Parallel Processing for Artificial Intelligence, pages 40-51, 1995. 15. S. S. Nerur. A hybrid parallel IDA* search. In Proceedings of the National Conference on Artificial Intelligence, 1994. 16. S. S. Nerur and D. J. Cook. A hybrid parallel-window/distributed tree algorithm for improving th e performance of search-related tasks. In Proceedings of the Sev-

enth International Conference on Industrial and Engineering Application of Artificial Intelligence and Expert Systems, pages 629-637, 1994. 17. J. Pearl. Heuristics. Addison-Wesley, Reading, Massachusetts, 1984. 18. C. Powley, C. Ferguson, and R. E. Korf. Parallel tree search on a simd machine. In Proceedings of the Third IEEE Symposium on Parallel and Distributed Processing, pages 249-256, 1991. 19. C. Powley, C. Ferguson, and R. E. Korf. Depth-first heuristic search on a simd machine. Artificial Intelligence, 60(2):199-242, 1993. 20. C. Powley and R. E. Korf. Single-agent parallel window search. IEEE Transactions on Pattern Analysis and Machine Intelligence, 13(5), 1991. 21. V. N. Rao, V. Kumar, and K. Ramesh. A parallel implementation of IterativeDeepening-A*. In Proceedings of AAAI, pages 178-182, 1987. 22. C. B. Suttner. Static partitioning with slackness. In Proceedings of the IJCAI Workshop on Parallel Processing for Artificial Intelligence, pages 143-155, 1995. 23. J. Xu and K. Hwang. Heuristic methods for dynamic load balancing in a messagepassing multicomputer. Journal of Parallel and Distributed Computing, 18(1):1-13, 1993.

143

Diane Cook Diane Cook is currently an Associate Professor in the Computer Science and Engineering Department at the University of Texas at Arlington. Her research interests include parallel algorithms for artificial intelligence, machine planning, and machine learning, and she has published over 60 peer-reviewed journal articles, book chapters, and conference papers in these areas. Dr. Cook received her B.S. from Wheaton College in 1985, and her M.S. and Ph.D. from the University of Illinois in 1987 and 1990, respectively. Her current Home Page: http://www-cse.uta.edu/'cook/home.html

Parallel Processing for Artificial Intelligence 3 J. Geller, H. Kitano and C.B. Suttner (Editors) 1997 Elsevier Science B.V.

145

Static Partitioning with Slackness Christian B. Suttner a* aInstitut fiir Informatik TU Miinchen, Germany Email: [email protected] Static Partitioning with Slackness (SPS) is a method for parallelizing search-based systems. Traditional partitioning approaches for parallel search rely on a continuous distribution of search alternatives among processors ("dynamic partitioning"). The SPSmodel instead proposes to start with a sequential search phase, in which tasks for parallel processing are generated. These tasks are then distributed and executed in parallel. No partitioning occurs during the parallel execution phase. The potentially arising load imbalance can be controlled by an excess number of tasks (slackness) as well as appropriate task generation. The SPS-model has several advantages over dynamic partitioning schemes. The most important advantage is that the amount of communication is strictly bounded and minimal. This results in the smallest possible dependence on communication latency, and makes efficient execution even on large workstation networks feasible. Furthermore, the availability of all tasks prior to their distribution allows optimization of the task set not possible otherwise. The paper describes the basic SPS-model, presents general simulation results, provides a worst-case comparison with other parallelization approaches, and discusses the appropriateness of using the model for parallelization. 1. I n t r o d u c t i o n

Combinatorial search (or simply search) denotes the quest for a solution to a problem by systematic trial-and-error. It is a fundamental computational paradigm that can be applied to many problems where the sequence of steps required to obtain a solution is not known in advance. Especially the fields of Artificial Intelligence and Engineering involve many problems where this is the case. Unfortunately, as the minimal number of steps (in terms of ideal search decisions) required for the solution of a problem increases, the size of the associated search space typically grows exponentially or even faster (combinatorial explosion). This rapidly decreases the chances to find solutions for non-trivial problem instances. Therefore, much work is devoted to the improvement of the search capabilities of current systems. A particularly promising approach is parallelization. Parallel search allows a significant increase in the utilized processing power, making it possible to search orders of magnitude larger search spaces. Moreover, while breadth-first search usually is *This research has been funded by the DFG SFB 342 subproject A5 (Parallelization in Inference Systems).

146 infeasible due to the exponential growth in memo .ry requirements, the concurrent exploration of different search alternatives can be used to introduce a breadth-first component into depth-first search which avoids the detrimental effects of wrong search decisions at an early stage of the search, which can result in exponential reductions of the required amount of search. The traditional approach for parallelizing search is to partition the search space in a dynamic fashion. This means that different search alternatives are processed in parallel, and newly arising alternatives are more or less frequently distributed among processors (e.g., [5,1,13]). The advantage of this is that the partitioning can automatically adapt to the search space. Idle processors can receive new work from other processors, and thereby a dynamic load balancing capability is available. However, there are also disadvantages. Dynamic partitioning allows no global control of the partitioning process, and the tasks 2 generated cannot be easily treated with respect to their global priority (because there is no global view). Furthermore, the realization of work migration for dynamic load balancing may result in extensive communication, increasing the parallelization overhead significantly. Finally, dynamic partitioning capabilities in practice are limited due to finite task buffers. In this paper, a parallelization model is proposed in which no partitioning occurs during the parallel computation. Instead, the partitions required for parallel execution are determined in a separate search process, at the beginning. In order to avoid load imbalance, the number of generated partitions may exceed the number of processors (slackness). For distinction from dynamic partitioning, this scheme is called static partitioning with slackness. The application of this model to the parallelization of a high performance sequential theorem prover has lead to substantial performance improvements. A detailed and extensive analysis in [17] shows that significantly more problems become solvable, and good speedup results are achieved. Improvements are also obtained compared to a previous parallelization of the same sequential system, which is based on the random competition model [6]. In the paper here, the general SPS model is presented, and some of the results on its general performance (i.e., independent of a particular application) are given. Section 2 contains a description of the basic model, and discusses the consequences of the model phases for a parallelization. Section 3 treats specifically the load balance issue, containing general simulation results. Section 4 provides a worst-case discussion of the SPS-model compared to two other parallelization approaches. In Section 5, the appropriateness of using the SPS-model is discussed. Section 6 gives an overview of related work. Finally, Section 7 contains a summa .ry of the presented material.

2. P h a s e s of t h e S P S - M o d e l Informally, the execution of a parallel search-based system based on the SPS-model can be separated into three phases as shown in Figure 1. 2We use the term task to denote a subproblem of the original problem. The processing of a task leads to the exploration of some part of the search space of the original problem.

147

Phase 1 Task Generation

1 Phase 2 Task Distribution

1 Phase 3 Parallel Execution

Figure 1. The phases of Static Partitioning with Slackness.

2.1. T h e Task G e n e r a t i o n P h a s e The task generation phase is defined by a set of partitioning rules, a generation strategy, and the desired number of tasks. An initial, finite segment of the search space is explored according to the generation strategy (e.g., depth-first, breadth-first, or best-first search). Depending on the strategy, the search is abandoned at certain points (e.g., by invoking backtracking due to a search bound), and the search states constituting unexplored alternatives are stored as tasks for later parallel execution. The type of tasks generated depends on the employed partitioning rules. Choices for partitioning rules are AND- and OR-partitioning. For AND-partitioning, a given set of subgoals is split such that each subgoal forms a separate task. Each of these AND-tasks needs to be solved to provide a solution to the original problem, and the solutions for the AND-tasks need to be compatible. A special case occurs for independent-AND partitioning: if the individual subgoals are independent of each other (e.g., do not share a variable directly or indirectly), incompatible solutions cannot occur and the subgoals can be solved independently without a final compatibility check. For OR-partitioning, different search alternatives are used to produce a set of OR-tasks, and the solution to any of these tasks is sufficient for a solution to the original problem. The desired number of tasks m is equal to or larger than the number of processors n to be utilized. The term slackness is used to express the relation between the number of tasks and the number of processors. In order to specify a degree of slackness, the slackness m shall denote the average number of tasks (slackness) per processor. parameter spp = -~ C o n s e q u e n c e s for Parallelization The advantages and disadvantages due to an isolated task generation phase before a parallel execution phase can be summarized as follows: 9 Information about the problem-specific search space becomes available. Such information is of high heuristic significance, and can be used to control many aspects of a search-based system. In particular, it allows

148 the identification of problems inadequate for parallelization. Untypically low branching factors or large numbers of search tree nodes without successors may indicate inappropriateness for parallel treatment or suggest that only some fraction of the available processors should be used. Since the initialization of a parallel execution on many processors often causes significant overhead, restricting the virtual machine size or refraining from parallel execution completely for such problems can actually reduce the execution time as compared to a full scale parallelization. Proper treatment of this type of problems is important for achieving good average case performance for applications where no pre-selection of especially hard problems (i.e., problems on which other attempts have previously failed) is done.

-

- redundancy control and significance-driven search space partitioning. In case (independent-)AND-partitioning is employed, it is possible to inspect the generated task set for redundant (e.g., syntactically equivalent) tasks. Due to such analysis, redundant search can be avoided and tasks which are important due to their frequent occurrence can be partitioned further, leading to a significance-driven task generation control. - a dynamic adjustment of system control parameters. Depending on the number of search tree nodes without successors encountered during the generation phase, the potential for load imbalance can be estimated. This can be used to adjust the degree of slackness appropriately. Furthermore, many search-based systems are controlled by a number of parameters. Based on information gained during generation, these parameters can be adjusted in a problem dependent way, for the parallel execution phase (e.g., the bound increment for systems using iterative-deepening search). 9 No parallelism is utilized in the initial phase. The effect of this on the performance depends on the following issues: - For large search problems, the duration of the generation phase is negligible. - In the generation phase, there is often little parallelism available (the search space just starts to "explode"). Parallel treatment of this phase may therefore be of limited effectiveness. - There is no search rate slowdown due to parallelization management (such as communication for task management). -

Within a given computing environment, the fastest available sequential hardware can be utilized, which can be many times faster than the average processor available for a parallel computation.

- Simple problems are solved immediately sequentially (no burden is put on a parallel computing environment); in those cases a parallel computation may require more time than a sequential one due to communication overhead. 2.2.

The

Task

Distribution

Phase

In the task distribution phase, the previously generated tasks are distributed among the available processors. The availability of all tasks prior to their distribution can be utilized

149

beneficially in several ways. First, redundancies among the tasks can be investigated and removed. Second, relationships among the tasks can be used to guide the mapping onto processors, e.g., to reduce the potential for load imbalance. Third, the complete mapping information can be given to each processor. This allows direct processor-toprocessor communication for distributed task management (such as termination control), or in case information exchange between particular tasks is desired (for extensions of the basic SPS-model). Regarding an ideal mapping of tasks to processors, information about the search space (task size) would be required. Unfortunately, such information is usually not available. In that case a good heuristic is to maximize the distance among tasks mapped to the same processor with respect to the number of common ancestors in the search tree. The motivation for this is that tasks which share many common ancestors denote similar situations, and are more likely to span search spaces of similar size than tasks with less commonalities. Indeed, the results in [9] show that such a heuristic performs best among a set of mapping strategies. A simple approximation to this heuristic is easy to realize: Each processor obtains eve .ry n-th task from a task list in which the tasks are ordered in the sequence of their generation (i.e., Taski ~ P(i ,nod n)+l, where Task~ denotes the i-th task generated, i E {1..m}, and Pj denotes processor j,

j e {1..~}).

2.3. T h e Parallel Task E x e c u t i o n P h a s e Finally, in the task execution phase, the tasks are executed independently of each other on their assigned processors. Since usually more than one task has to be processed per processor, a service strategy is required. Possible service strategies are serial execution and processor sharing. For serial processing, one task after the other is executed until a success state is found or all tasks failed. For processor sharing, the processing capacity of a processor is split among the tasks via preemptive execution (e.g., using a round robin strategy, where a so-called processing time-slice is given to one task after the other in a cycle until the computation terminates). Processor sharing is preferable over serial processing, because a task may not terminate within a given runtime limit. In applications involving undecidable search problems (such as automated theorem proving), termination after a finite amount of time cannot even be guaranteed in principle. In that case, serial processing causes incompleteness of the search, since a nonterminating task would infinitely delay all tasks remaining at the respective processor. Even for decidable search problems, a strategy which is fair in the sense that the available runtime is evenly split among the tasks is desirable. Since in practice the search is bounded by a given runtime limit T,,nit, a fair strategy ensures that a solution is found if some task can be solved within T, mit/spp. Altogether, for each task to be processed on a processor, a copy of the original sequential search-based system (extended by the ability to process a task) is started under a time-sharing regime. It is noteworthy that no change of the main search engine of a sequential system is required for SPS-parallelization. All that is required is an interface which allows the processing of a given task (task generation can be done by a different system, or a specifically modified version of the sequential system). After initialization of the task processing, there is no slowdown of the search engine due to program code required for the parallelization.

150 3. L o a d B a l a n c e

In this section we inspect the effect of slackness on the load balance. For measuring the degree of load imbalance quantitatively, we define LI(n) (for Load Imbalance) as

LI(n) = E~=~(Tpe(,) - T~) ( n - 1) • Tpe(n)' where T/denotes the total runtime spent at processor i and Tp~(n) denotes the total system runtime (i.e., Ti < Tpe(n)), for n > 1. The term n - 1 in the denominator represents the largest number of terms which can be different from zero, because there is at least one processor i with T/ = Tpe(n) (Tpe(n) = max,=l...nTi). LI(n) is an absolute measure, not taking into account the best or worst possible balance that can be obtained for a particular set of tasks. It ranges from perfect balance (Ll(n) = 0), which means that all processors finish working at the same time, to maximal imbalance (LI(n) = 1), where exactly one processor is busy during the execution. In order to assess the effects of slackness, a general modeling of search is used. For the simulations, OR-partitioning is assumed. This means that a given problem is solved as soon as one of the tasks is solved. Also, it is assumed that m = n • spp, and that each processor obtains spp tasks. Each computation (consisting of the treatment of a set of tasks) is assumed to be constrained by a user determined runtime limit Zlimi t. Therefore, a computation terminates as soon as a task terminates successfully, or after all tasks failed, or when Zlimi t is reached, whichever occurs first. T h e probability of a task terminating successfully is specified by p (i.e:, a task terminates unsuccessfully with probability 1 - p). The runtimes for all tasks ai'e independently drawn from a uniform distribution which is equal to 0.5 from 0 to 2 and equal to 0 otherwise, resulting in a mean value of 1. Previous experiments [18] have shown little qualitative dependence of the load balancing effect on the particular choice of runtime distribution. The runtime limit is important for the simulation for two reasons. First, externally triggered termination by a runtime limit influences load balance. Early system termination (compared to the average runtime of a task) renders load imbalance unlikely. In the simulation, all tasks have a mean runtime of one unit of time, and runtime limits are issued as multiples of the mean. Second, the actual runtime of a task which is terminated by Zlimi t becomes irrelevant. Such a task might as well have an infinite runtime. Thus, in those cases where tasks are terminated due to Zlimit, the runtime limif allows the extrapolation of the results to distributions which have larger runtimes (for those tasks) and therefore have a larger variance. The system model does not take into account communication or time-sharing overhead. This omission, however, is unlikely to affect the qualitative results: Neglecting the communication overhead is tolerable since communication occurs only at the beginning (distribution of tasks) and at the end (collection of results and global termination)of the parallel computation, and thus mainly depends on the number and physical size of tasks, but not the duration of their computation. The time-sharing overhead is usually low for contemporary operating systems, as long as no swapping occurs. Thus, all processes running on the same processor should fit into the main memo .ry, thereby limiting the degree of slackness. Another limitation for the degree of slackness arises from the time-sharing delay for a task. For these reasons the investigated slackness is limited to 16.

151 Three different modeling variants are presented. There are two options for obtaining different slackness values for a parallel search. The first is to generate different numbers of tasks for a fixed number of processors, and the second is to choose different numbers of processors for a fixed number of tasks. In the first case, another important distinction arises: How does the increasing degree of partitioning influence the overall probability to find a solution (within the given time limit)? In practice, increasing partitioning is likely to increase the overall probability for success. This is modeled by assuming that the probability of success per task p remains constant for different slackness values (i.e., for different numbers of tasks). For the assumption that increasing partitioning does not improve the overall success probability, the value of p is decreased for increasing slackness such that the overall success probability remains constant (and equal to the value obtained for s p p - 1). The load imbalance values for these two options are shown in Figure 2. Figure 3 shows the results for the case where the number of processors is varied instead of the number of tasks. The load imbalance for s p p = 1 is the same in all plots, because in this case in all plots 32 processors and 32 tasks are used. All figures give results for a low success probability p = 0.01 (higher success probability values result in lower load imbalance, since the computation terminates earlier).

p overall(success), 0.01 0.45

............ .

.

.

.

.

i .

.

.

.

.

.

.

A .

.

. .

0.4

p(success), 0.01 . . . . r,~p ,,,- 1 . - , ~ | s p p - 2 --,--

0.35

0.45

0.35

t;, ........

0.3

0.3

777 ....... .....7 .......... 7-

0.25

0.25

0.2

0.2

0.15

/j._

0.1

i ....... -

0.05

0.15 0.1 0.05

.

0 0

5

.

.

,

.

I0

15

T limit

20

25

30

0 0

5

10

15 TJi~

20

25

30

Figure 2. Load imbalance LI for uniform runtime distribution for constant (left plot) and increasing (right plot) overall success probability.

The left plot in figure 2 shows that even in the (worst) case that no advantage is gained by increased partitioning, the load imbalance can be cut down by more than one half with a slackness of 16 tasks per processor. A much larger reduction occurs in the right plot, where L I becomes negligible for s p p >_ 8. This is due to the increasing overall~ success probability, which increases the chances for an early OR-parallel termination. The load imbalance reduction found in Figure 3 is about in between the two previous cases. The experiments show that for all modeling variants, slackness leads to a noteworthy reduction of the load imbalance.

152 p(succem) ,, 0.01

~ 0.4

0.45

0.3 ~176 0.2 0.15~ 0.1 0.05 00

5

.........................................

10

15 T_,mn

20

25

30

Figure 3. Load imbalance LI for uniform runtime distribution with 32 tasks. Different slackness values are obtained by varying the number of processors from 2 to 32.

In [18] a set of experiments regarding slackness has been reported, focusing on the case where the overall success probability increases with increasing slackness. They show that similar results are obtained for quite different runtime distributions (exponential, uniform, and triangle). This suggests that the load imbalance reduction of slackness is largely independent of the shape of the distribution. Those experiments also show that for success probabilities p > 0.01, small slackness values soon reduce load imbalance to negligible values.

4. Worst Case Analysis Regarding worst-case behavior, it may be suspected that an extremely unbalanced search space will render the SPS-model inappropriate as compared to a dynamic scheme, which can adapt to such a situation by dynamically producing new tasks as necessa~.. Although this may be the case in many specific situations, the following considerations surprisingly reveal that the SPS-model performs quite well compared to dynamic partitioning schemes, in some straightforward worst case scenarios. In the following, two general worst case situations are described and analyzed. In the first situation it is assumed that regardless of the parallelization method, the generation of tasks is always possible as desired, but a maximal work imbalance among the tasks occurs. In the second situation no parallelism is inherent in the problem. All discussions are based on the assumption that no solutions exist. This is necessa .ry to make a comparison between the approaches possible. If solutions are allowed, the performance of a parallelization approach depends critically on the number of solutions and their location in the search space (in relation to the parallel search strategy). Situation 1: Maximal Work Imbalance among Tasks. Regardless of the parallelization model employed, assume that at any time of the parallel computation, all but one of the

153 currently available tasks terminate after one search step. The particular task that does not terminate spans the remaining search space, and may be used to generate new tasks depending on the parallelization model. Regarding runtime and accumulated processing time, this scenario describes the worst possible situation that can occur for the SPS-model. It leads to the generation of m ---n • spp tasks (where n is the number of processors), which are distributed once among the processors. Thus, a runtime delay of O(n) (for processor-to-processor communication, assuming that spp tasks fit into one message package) and an accumulated processing time overhead of O ( m ) (for communication and task handling) occurs. As a benefit, n • spp search steps are performed in parallel. Assuming that a single search step takes much less time than distributing a task, the overhead will usually outweigh the benefit from parallel execution. Furthermore, the main work remains to be done by a single task (no further partitioning occurs after the first task generation3). Assuming the search terminates after k search steps (with k >> m), the constant amount of work performed in parallel will be insignificant, and therefore the runtime will be approximately the same as without parallelization. However, it is important to note that no adverse effects (besides the overhead for task generation and distribution of O ( m ) ) occur either. In particular, the single remaining task can be executed without slowdown arising due to the parallelization. Thus, while no speedup is achieved, no relevant slowdown occurs either. The increase in the accumulated processing time depends on the ratio between the time for initializing the processing of a task and the time for performing a single search step. Assuming a low initialization overhead, the accumulated processing time will remain close to the sequential runtime. Let us now turn to the behavior of dynamic search space partitioning approaches. A dynamic partitioning scheme has provisions to generate tasks (or obtain tasks from other processors) whenever some number of processors becomes idle. Thus, in the given scenario, all processors will continue to participate in the computation until termination. Therefore, in contrast to the SPS-model, the number of search steps executed in parallel is not constant, but increases as the search continues. This, in fact, is usually considered as the prima .ry advantage of dynamic partitioning compared to static partitioning. While of course there are situations where this ability will pay off, in the given scenario this is in fact disadvantageous: 9 There is a permanent need for communication. Depending on the parallel hardware platform, the frequent necessity for communication can significantly decrease the performance of the system. In a multiuser system, this can seriously affect other users as well. 9 Assuming that a communication operation together with the task processing initialization takes significantly longer than the execution of a single search step (a realistic assumption for most search-based systems), a large fraction of the accumulated processing time is spent on overhead rather than useful search. 3Note that in the SPS-model parallel execution is avoided in situations where the problem does not provide enough inherent parallelism to generate the desired number of tasks. This advantage is ignored in the analysis.

154 9 There is no single task which runs undisturbed; unless a specific task leads to the generation of all other tasks, fast processing of such a "main task" is not possible. In the described scenario, typical dynamic partitioning schemes will actually run longer than a SPS-based system, for at least n times the cost of a SPS-based system. Both T1 where Tap(,,) = F~'~=IT~ regarding speedup (S(n) = Tpe(, T~ ) ) and productivity9 ( P ( n ) = Tap(n) , is the accumulated processing time of all involved processors), the described scenario is significantly less harmful for the SPS-model than for dynamic partitioning schemes. In particular, the potential for negative effects of dynamic partitioning schemes in multiuser time-sharing systems require precautions which, in effect, can only be achieved by reducing the dynamic partitioning ability of the system, thereby moving towards a static model. Of course, scenarios where dynamic partitioning performs better than the SPS-model exist. For example, if most of the tasks that are generated by the SPS-model terminate immediately unsuccessfully, and the remaining tasks could be partitioned further into equal slices of work, a dynamic partitioning scheme would be advantageous. In fact, this particular situation maximizes the advantage of dynamic partitioning over static partitioning. Altogether, the performance of a parallelization scheme depends not only on the structure of the search space. The size of tasks, the relationship of communication time to task runtime, and the given runtime-limit all influence the adequacy of a partitioning approach, and make an absolute comparison between static and dynamic schemes difficult. The advantages of static partitioning over dynamic partitioning are mainly due to the initial exploration phase lying at the base of the SPS-model. Of course, one may argue that such a phase can be used at the beginning for any dynamic partitioning scheme as well, combining the best of both approaches. This, indeed, can lead to interesting systems. A simulation study which investigates the use of an initial task distribution for a dynamic partitioning scheme is found in [15]. In this study, the use of an initial task distribution increased the efficiency Erei by approximately 15~ when more than about 50 processors were used. It thereby improved the scalability of the employed dynamic partitioning scheme. In general, it is difficult to determine in which cases the additional implementation effort and computational overhead for dynamic partitioning pay off. In practice the unnecessa .ry employment of dynamic partitioning may lead to increased computational costs for little or no benefit (if not slowdown).

Situation 2: No Inherent Parallelism. The worst case with respect to distributing work among processors is that the search tree consists of a single path (i.e., no alternatives for search steps exist). Thus, neither static nor dynamic partitioning is possible. Static partitioning simply performs sequential search in that case, since no parallel tasks are generated. Assuming an appropriate design of the task generation phase, the overhead for potential task generation is negligible. The performance of dynamic partitioning approaches depends on the particular scheme employed. If parallelism is utilized only after an initial set of tasks has been generated, no parallelization will occur and the performance is comparable to the SPS-model. Otherwise, if parallel processes are started independently of the current availability of tasks, a significant overhead may occur.

155

Performance Stability. Another important issue regarding performance is its stability. In multiuser systems the time for a communication operation is not fixed, but depends on the interconnection traffic caused by other users. Similarly, the load of individual processors may change due to processes unrelated to the intended computation. Both factors considerably influence the communication timing (i.e., the order in which communication events occur). In the SPS-model, a communication delay in the worst case causes a prolongation of the computation and an increase of the computational costs both of which are bounded by the order of the delay. The reason that the change is bounded is that the search space partitioning is independent of the order in which the communication events occur. In dynamic partitioning schemes, however, the generation of tasks, and therefore the exploration of the search space, is usually dependent on the order in which tasks are processed. As a consequence, large variations in the runtime may occur (see for example [12]). In general, changes in the communication overhead will lead to an undesirable system operation mode for dynamic partitioning approaches, because such systems are usually tuned to optimize their performance based on knowledge about typical communication performance parameters (for work on such optimization for a parallel theorem prover, see [10]).

Summary of Worst Case Considerations. The fact that in many particular situations dynamic partitioning schemes provide better flexibility for adapting to irregularly structured search problems is obvious. However, the previous discussions show that the overhead incurred with this flexibility leads to a nontrivial trade-off between static and dynamic schemes which is frequently overlooked. In general, the following statements can be made. Disadvantageous scenarios for the SPS-model lead to a strictly limited overhead with acceptable upper bounds. If the worst case occurs, there is no benefit from parallel computation. However, as the possible speedup decreases for more unfortunate situations, the accumulated processing time decreases as well. The computational gain (product of speedup and productivity) for the SPS-model achieves acceptable values even in worst case scenarios, for problems of sufficient size. This is not necessarily the case for dynamic partitioning schemes, for which the worst case overhead cannot be bounded easily. A benefit from parallel computation may not only be lost, it may actually lead to a significant performance decrease for this and, in multiprogramming environments, other computations. This happens because, unlike for the SPS-model, the accumulated processing time increases. As a result the computational gain can drop to very low values in the worst case for dynamic partitioning approaches, regardless of the problem size.

5. Appropriateness of the S P S - M o d e l This section consists of three parts: a discussion of the SPS-model with respect to important design issues arising for the construction of a parallel system; a list of system properties which make the application of the SPS-model particularly interesting; and a summary of the advantages and disadvantages that can arise from using the SPS-model.

156 5.1. D i s c u s s i o n of Suitability In general, the adequacy of a particular parallelization approach depends on many issues. A decision among different approaches requires a detailed specification of the intended goal of parallelization. In order to specify the intended use of parallelism sufficiently, at least the issues described below need to be clarified. For each item, first a discussion regarding parallelism in general is given, followed by remarks regarding the SPS-model. 9 W h i c h t y p e of c o m p u t i n g problems are to be solved? In general: Performance can be optimized for the average case or for specific problems. Especially for parallel systems, this distinction makes a significant difference. A large enough problem size commonly leads to a good scalability of most techniques, and therefore tends to pose little difficulties. The treatment of comparatively small problems, however, often leads to unacceptable overheads due to the unprofitable initialization of a large parallel system. Since the size of a search problem usually is not known prior to its solution, this can result in the inadequacy of a parallelization technique if average case performance improvement is desired. SPS-model: The SPS-model avoids parallelization overhead for problems which are small enough to be solved during the task generation phase. This feature automatically adapts to the machine size: more processors require more tasks, which leads to more search during task generation; thereby more problems become solvable before parallel processing is initiated (in effect, problems need to be more difficult in order to be parallelized on larger machines). Furthermore, it is possible to determine heuristically the number of processors to be utilized, based on information about the search space growth gathered during the task generation phase. It can thereby support the processing of medium-size problems, keeping the initialization costs at a level suitable for the given problem. 9 W h i c h t y p e of parallel m a c h in e will be used? In general: The particular strengths and weaknesses of the intended hardware platform (e.g., memory model, topology, communication bandwidth) significantly influence the suitability of a parallelization technique. Techniques with little requirements on hardware properties are less sensitive to this issue, while some parallelization approaches can be realized efficiently only on specific hardware platforms. SPS-model: The SPS-model has particularly little communication and no memo.ry model requirements, and is therefore suited to all types of MIMD machines, including workstation networks. 9 W h a t is th e intended degree of portability to different parallel ma c hi ne s ? In general: If no portability is required, an implementation can be optimized for combining the parallel model and the hardware. However, such tight bounds limit the lifetime of the system severely. Due to the unavailability of a common machine model for parallel computers (such as the von-Neumann model for sequential computers), successor models often feature major design changes. In that case, a specifically tuned implementation is bound to its first platform, and may be soon outperformed by an improved sequential system on the latest sequential hardware.

157

SPS-model: Due to its little requirements on communication performance, the SPSmodel can be realized efficiently using portable implementation platforms, such as PVM [2], p4 [4], or MPI [7]. This makes an implementation available on a large number of parallel machines 4 as well as on workstation networks. 9 W h a t is t h e m i n i m a l p e r f o r m a n c e increase e x p e c t e d ? In general: A given desired increase (e.g., S > 100) constrains the minimal number of processors, and thereby defines a minimal degree of required scalability. Scalability of search-based systems is application-specific, and can be hard to predict for a particular parallelization method. SPS-model: The SPS-model simplifies scalability prediction and analysis due to the possibility of simple and meaningful sequential simulation before a parallel system is built. 9 W h a t is t h e desired trade-off b e t w e e n s p e e d u p and p r o d u c t i v i t y ? In general: The adequacy of a parallelization technique depends on the relative importance of speedup S T1 and productivity P = T1 5 This can be %e(,) " Tap(,)" expressed by choosing an appropriate tuning parameter r in the definition of a computational gain G = P x S r, r E ]R+. SPS-model: Taking into account the accumulated processing time has been one of the driving motivations for the development of the SPS-model. The static task generation avoids much of the communication and management overhead required for dynamic partitioning schemes. 9 Are t h e r e r e q u i r e m e n t s regarding worst case p e r f o r m a n c e ? In general: The worst case runtime and accumulated processing time vary significantly for different parallelization approaches, and depend heavily on a number of system aspects. SPS-model: As shown in Section 4, the SPS-model has better worst case performance than many other approaches. 9 D o e s t h e search-based s y s t e m d e p e n d on i t e r a t i v e - d e e p e n i n g search? In general: Iterative-deepening is a wide-spread search technique. For parallel systems employing this method, the maintenance of deepening balance is desirable. Dynamic partitioning schemes in principle allow control of the balance, however for impractical costs (the associated communication problem is NP-complete). SPS-model: In [17] it is shown that slackness together with an iterative-deepening diagonalization and a delayed successor start strategy are effective techniques for reducing deepening imbalance (i.e., the differences in the iterative-deepening levels worked on at different processors) and load imbalance, without requiring explicit communication. 4E.g., PVM is available on: Intel iPSC/2, iPSC/860, and Paragon; Kendall Square Research KSR.-1; Thinking Machines CM2 and CM5; BBN Butterfly; Convex C-series; Alliant FX/8; etc. 5The formulas represent relative or absolute speedup and productivity, depending on the definition of 7'1. If T1 equals the runtime of the parallel system on one processor, relative metrics are obtained. If 7"1 equals the runtime of a sequential reference system, absolute metrics result.

158

9 A r e there s y s t e m - e x t e r n a l constraints? In general: In multiuser systems, constraints due to the computing environment arise. The number of processors available for computation may not be constant for all computations. This arises either when an independent system resource management splits a parallel machine into fixed partitions for individual users or when the load on individual nodes discourages additional processing load. SPS-model: It is possible to take such constraints into account within the SPSmodel, and to adjust the search partitioning to the available number of processors. 5.2. Beneficial P r o p e r t i e s for A p p l i c a t i o n There are several properties of search-based systems which render them suitable for the application of the SPS-model: 9 low probability of load i m b a l a n c e Tasks which only span a small search space (and do not contain a solution) cause load imbalance and therefore should be rare. This may be known empirically for an application, or may be ensured for individual tasks by look-ahead during task generation. 9 fast task g e n e r a t i o n A fast task generation reduces the serial fraction of the computation caused by the task generation phase. Useful for this are - a high task generation rate; Only a small amount of search is required to produce the next task. - an efficient task representation. Tasks can be stored and transferred with little overhead. This is generally desirable for all parallelization schemes, because it reduces the parallelization overhead.

5.3. S P S - M o d e l : C o n s e q u e n c e s of its A p p l i c a t i o n A summary, of the consequences of the application of the SPS-model is given below. Appropriate usage of the model promises the following properties: 9 little c o m m u n i c a t i o n is required As a consequence, - the communication overhead is bounded and small. This is important for achieving good productivity. - there are little requirements on hardware communication performance. Therefore, the SPS-model is well suited to the usage of general purpose parallel programming libraries and networks of workstations. - the complexity of communication is low, which simplifies the implementation and maintenance effort required.

159

9 informed d y n a m i c decisions about parallelization and search Based on information gathered during the task generation phase, heuristic decisions can be made regarding parallelization (e.g., appropriate number of processors, appropriate slackness) and search control (e.g., appropriate iterative-deepening increments). See Section 2.1. 9 global search o p t i m i z a t i o n The use of AND-partitioning can lead to a reduction of the amount of search to be done (see Section 2.1). 9 little m o d i f i c a t i o n of the target s y s t e m The search part of a sequential system does not need to be modified. The necessa .ry extensions consist of a means to produce tasks, and the ability to start the search given by a task. 9 efficient integration of A N D - p a r a l l e l i z a t i o n The use of static task generation before parallel execution allows control over the overhead induced by AND-parallelism. 9 m e a n i n g f u l s i m u l a t i o n for any n u m b e r of processors This is possible because the parallel exploration of the search space does not depend on the communication timing. For a simulation, all generated tasks can be processed in a sequence. The results can be combined to obtain results for any slackness between s p p - 1 (number of processors n - m) and s p p - m ( n - 1).

9 c o m b i n a t i o n of different search-based s y s t e m s Different systems can be combined by using one system for task generation, and several copies of one or more different systems for task processing. In particular, it allows a combination of forward- and backward-chaining search. In cases where the SPS-model is inappropriate, the following consequences of its application may occur:

9 no s p e e d u p c o m p a r e d to the original sequential s y s t e m (worst case) However, no significant slowdown occurs either (see also Section 4). 9 th e task g e n e r a t i o n phase b e c o m e s a b o t t l e n e c k For the generation of a large number of tasks, or in systems with a low task generation rate, the overall computation is slowed down due to the serial fraction caused by task generation. The potential for a bottleneck can be reduced by distributing tasks as soon as they become available and by hierarchical task generation.

9 th e task d i s t r i b u t i o n phase b e c o m e s a b o t t l e n e c k This can be avoided by distributing tasks during the generation phase (see previous item) or by hierarchical distribution. 9 a p e r f o r m a n c e discontinuity occurs Whenever the initial exploration phase finishes immediately before a solution could

160

be found by continued sequential search, the communication time to distribute the tasks and collect the results is wasted. In this particular case, a runtime increase compared to the sequential runtime occurs. The runtime decreases as the amount of further search required after the generation phase increases. 6. R e l a t e d W o r k Work related to the SPS-model can be grouped into three topics, namely research on static scheduling, usage of a task pool, and bulk synchronous programming. Static Scheduling. A traditional static scheduling problem is, given a set of independent tasks and the runtime for each, to find a distribution of the tasks such that the overall runtime is minimized. The tasks considered typically form an independent-AND-set, i.e., all given tasks need to be solved, but independence is assumed. This scheduling problem is well-known to be NP-complete [8], and research in this area focuses on efficient approximations to optimal solutions [3,16]. Unfortunately, for search-based systems the runtime is usually neither known nor can it be estimated accurately. Work on static scheduling without given knowledge about task runtimes is rare. However, interesting research with relation to the SPS-model is found in [11]. The authors investigate static task distribution for minimizing the runtime of a set of independentAND tasks. In their model, each processor repeatedly obtains a batch of k subtasks from a central queue and executes them, until all subtasks have been processed. For the case k - r n~, k becomes equivalent to spp. The authors conclude that for many distributions a static allocation provides reasonable performance relative to an optimal scheme. Task Pool. An alternative for the processor sharing of several tasks in the SPS-model is to choose a pool model: only one task is initially distributed to each processor, and the remaining tasks are stored in a central pool. Whenever a processor finishes its task unsuccessfully, it obtains another task from the pool. Obviously, such a scheme obtains a better load distribution due to its dynamic reactivity. For this, it requires additional communication and control. The expected performance of such a scheme for OR-parallel search has been theoretically analyzed in [14], for three different runtime distributions of tasks (constant, exponential, and uniform) and the probability of success as a variable. The case of constant runtime for all tasks (not realistic for search problems), in fact, is identical for the pool model and the SPS-model, if serial execution of the tasks at a single processor is chosen in the SPS-model. The pool model (as well as serial execution of tasks at one processor for the SPS-model), however, is inappropriate for many applications of search. The reason is that any practical search has to be terminated after some finite time, i.e., a runtime limit is posed onto each search process to ensure termination. For difficult search problems, many tasks take longer to terminate than can be allotted by any reasonable means. In fact, for undecidable problems termination itself cannot be guaranteed. Thus, in a model of computation where some tasks are delayed until some other tasks terminate, the ve.ry tasks which may allow a solution to be found quickly (which is the spirit of OR-partitioning) may be executed prohibitively late (or even never). Therefore a pool model is inappropriate. Bulk Synchronous Programming. The SPS-model also bears some relation to the bulk synchronous programming (BSP) model developed by L. Valiant [19,20]. In this model,

161 the programmer writes a program for a virtual number of processors v, which is then executed on a machine with n processors. According to the model, n should be much smaller than v (e.g. v = n log n). Then thisslackness can be exploited by compilers in order to optimize scheduling and communication. As for the SPS-model, a surplus of tasks is used to achieve a load-balancing effect. However, the BSP model is intended as a basic computational model for parallel processing, and assumes compiler and operating system support. It allows communication and dependencies between the tasks, and assumes that all tasks need to be finished for completing a job (AND-parallelism). While the BSPmodel is a model for general computations, the SPS-model is focused specifically towards search-based systems, where more specific assumptions apply.

7. S u m m a r y In this paper, the parallelization scheme static partitioning with slackness has been presented, independent of a particular application. The advantages and disadvantages of a sequential initial search phase for task generation have been discussed. The potential drawback of the model, namely the occurrence of load imbalance due to tasks with finite (and small) search spaces, can be effectively reduced by slackness. A worst case analysis revealed that, unlike for other parallelization approaches, the worst case for the SPS-model is bounded and moderate. Typical design issues occurring for the construction of a parallel search-based system have been considered; then advantageous system properties for applying SPS parallelization and resulting properties of a parallel system were presented. Finally, research related to the SPS-model has been discussed.

REFERENCES 1. K.A.M. Ali and R. Karlsson. The MUSE Or-parallel Prolog Model and its Performance. In Proceedings of the 1990 North American Conference on Logic Programming. MIT Press, 1990. 2. A. Beguelin, J. Dongarra, A. Geist, R. Manchek, and V.S. Sunderam. A User's Guide to PVM: Parallel Virtual Machine. Technical Report ORNL/TM-11826, Oak Ridge National Laborato .ry, 1991. 3. K.P. Belkhale and P. Banerjee. Approximate Algorithms for the Partitionable Independent Task Scheduling Problem. In Proceedings of the 1990 International Conference on Parallel Processing, 1990. 4. R. Butler and E. Lusk. Users Guide to the p4 Programming System. Technical Report ANL-92/17, Argonne National Laboratory, 1992. 5. W.F. Clocksin and H. Alshawi. A Method for Efficiently Executing Horn Clause Programs using Multiple Processors. New Generation Computing, (5):361-376, 1988. 6. W. Ertel. Parallele Suche mit randomisiertem Wettbewerb in Inferenzsystemen, volume 25 of series DISKI. Infix-Verlag, 1993. 7. Message Passing Interface Forum. MPI: A Message-Passing Interface Standard. 1994. 8. M.R. Garey and D.S. Johnson. Computers and Intractability: A Guide to the Theory of NP-Completeness. Freeman, 1979.

162 M. Huber. Parallele Simulation des Theorembeweiser SETHEO unter Verwendung des Static Partitioning Konzepts. Diplomarbeit, Institut fiir Informatik, Technische Universit/it Miinchen, 1993. 10. M. Jobmann and J. Schumann. Modelling and Performance Analysis of a Parallel Theorem Prover. In A CM SIGMETRICS and PERFORMANCE '92, International

Conference on Measurement and Modeling of Computer Systems, Newport, Rhode Islands, USA, volume 20, pages 2 5 9 - 260, SIGMETRICS and IFIP W.G. 7.3, 1992. ACM. 11. C.P. Kruskal and A. Weiss. Allocating Independent Subtasks on Parallel Processors. IEEE Transactions on Software Engineering, SE-11(10):1001-1016, 1985. 12. E. Lusk and W. McCune. Experiments with ROO, a Parallel Automated Deduction System. In Parallelization in Inference Systems, pages 139-162. Springer LNAI 590, 1992. 13. E.L. Lusk, W.W. McCune, and J. Slaney. ROO: A Parallel Theorem Prover. In Proceedings of CADE-11, pages 731-734. Springer LNAI 607, 1992. 14. K. S. Natarajan. Expected Performance of Parallel Search. In International Conference of Parallel Processing, pages 121-125, 1989. 15. J. Schumann and M. Jobmann. Analysing the Load Balancing Scheme of a Parallel System on Multiprocessors. In Proceedings of PARLE 94, LNCS 817, pages 819-822. Springer, 1994. 16. B. Shirazi, M. Wang, and G. Pathak. Analysis and Evaluation of Heuristic Methods for Static Task Scheduling. Journal of Parallel and Distributed Computing, (10):222232, 1990. 17. C.B. Suttner. Parallelization of Search-based Systems by Static Partitioning with Slackness, 1995. Dissertation, Institut fiir Informatik, Technische Universitht Miinchen. Published as volume 101 of series DISKI, Infix-Verlag, Germany. 18. C.B. Suttner and M.R. Jobmann. Simulation Analysis of Static Partitioning with Slackness. In Parallel Processing for Artificial Intelligence 2, Machine Intelligence and Pattern Recognition 15, pages 93-105. Elsevier, 1994. 19. L.G. Valiant. A Bridging Model for parallel Computation. Communication of the A CM, 33(8), August 1990. 20. L.G. Valiant. General Purpose Parallel Architectures. In J. Van Leeuwen, editor, Handbook of Theoretical Computer Science, chapter 18. Elsevier Science Publishers, 1990.

163 Christian S u t t n e r

Christian Suttner studied Computer Science and Electrical Engineering at the Technische Universit~it Miinchen and the Virginia Polytechnic Institute and State University. He received a Diploma with excellence from the TU Miinchen in 1990, and since then he is working as a full-time researcher on parallel inference systems in the Automated Reasoning Research Group at the TU Miinchen. He received a Doctoral degree in Computer Science from the TUM in 1995. His current research interests include automated theorem proving, parallelization of search-based systems, network computing, and system evaluation. Together with Geoff Sutcliffe, he created and maintains the TPTP problem library for automated theorem proving systems and designs and organizes theorem proving competitions. Home Page: http://wwwjessen.informatik.tu-muenchen.de/personen/suttner.html

Parallel Processing for Artificial Intelligence 3 J. Geller, H. Kitano and C.B. Suttner (Editors) 9 1997 Elsevier Science B.V. All rights reserved.

165

Problem Partition and Solvers Coordination in Distributed Constraint Satisfaction P. Berlandier a and B. Neveu b aILOG Inc. 1901 Landings Drive Mountain View, CA 94043, USA bINRIA - CERMICS 2004, Route des Lucioles, B.P. 93 06902 Sophia-Antipolis Cedex, FRANCE This paper presents a decomposition-based distributed algorithm for solving constraint satisfaction problems. The main alternatives for distributed constraint satisfaction are reviewed. An algorithm using a partition of the constraint graph is then detailed, with its parallel version. Experiments on problems made of loosely connected random constraint satisfaction problems show its benefits for under-constrained problems and problems with a complexity in the phase transition zone. 1. I n t r o d u c t i o n Many artificial intelligence problems (e.g. in vision, design or scheduling) may take the shape of constraint satisfaction problems (csPs) [1]. Being NP-complete, these problems are in need of any computational mean that could speed up their resolution. Parallel algorithms and parallel hardware are good candidates to help in this matter. A second motivation for distribution is that there exist csPs where the structure of the constraint graph is naturally close to a union of independent components. We are indeed especially interested in problems that result from the connection of subproblems by a global constraint. In such problems, the partition into subproblems is given and the challenge is to use that partition in a distributed search algorithm in order to speed up the resolution. Such problem structures happen to be quite common in configuration or design problems, where the whole problem consists of an assembly of subparts, each having its own constraints and being connected by a few global constraints on decision variables such as cost, weight or volume. In this paper, we will present a distributed algorithm for solving csPs over several processors using a partition of their constraint graph. 2. Sources of Parallelism in C S P R e s o l u t i o n The usual way of solving a csP alternates problem reduction and variable instantiation [2]. There are several opportunities for introducing some amount of parallelism in

166 these two processes. We give a brief review of these opportunities below. P a r a l l e l i s m in P r o b l e m R e d u c t i o n

Problem reduction is usually achieved by enforcing some level of partial consistency such as arc or path consistency [3] or by using a more limited filtering process such as forward checking [2]. Some operations that are required to enforce partial consistency can be performed independently. First, checking the consistency usually means controlling which possible value combinations are allowed by a constraint. These tests can obviously be conducted in parallel. This source of parallelism is easily exploited in most constraint systems by the use of operations on bit vectors [4]. A coarser grain parallelism is the parallel propagation of constraints: several constraints (or groups of connected constraints) are activated independently. Some synchronization mechanism is needed as different constraints might share the same variable. However, the fact that constraint propagation results in a monotonic reduction of the problem may simplify the synchronization. Parallel propagation [5-8] has received a great deal of attention. Especially, several procedures to achieve parallel arc-consistency have been devised, sometimes exhibiting a supra-linear speedup. However, [9] exhibits some theoretical restrictions on the gain that can be expected from this kind of parallelism. P a r a l l e l i s m in Variable I n s t a n t i a t i o n

Variable instantiation is a tree search process and the independent exploration of the different branches leads to or-parallelism. The introduction of or-parallelism in search procedures has been studied thoroughly, especially inside [10] but also outside [11] the logic programming community. An experiment of exploiting or-parallelism in the CHIP constraint logic programming language is described in [12]. P a r a l l e l i s m based on t h e C o n s t r a i n t G r a p h

Another way to parallelize the resolution is to partition the variable set and to allocate the instantiation of a subset of the variables to a process. The difficulty of this approach is to synchronize the different processes, which are not independent: there exist constraints between variables and conflicts may occur between the processes. A solution is to order the processes [13,14]. Another way to solve this difficulty is to have a central process that is responsible of the inter process conflict resolution. We will detail our approach in the next section. It takes place in that graph-based distributed framework, with a centralized control. 3. D i s t r i b u t e d C o n s t r a i n t Satisfaction

Binary constraint problems can be represented as graphs where variables are mapped to the vertices and constraints are mapped to the edges. For instance, the constraint graph associated with the well-known N-queens constraint problem is a clique: each variable is connected to all the others. But this is not the case for most real-world problems where it is more common to have loosely connected clusters of highly connected variables. Such

167

Constraint-connected subproblems

Variable-connected subproblems

Figure 2. Solving subproblems independently

almost independent subproblems could thus be solved almost independently with parallel processes and their results combined to yield a global solution. This is the approach proposed in the paper. The most important questions are: (1) How can the problem be partitioned "well"? (2) How can the efforts of the separate solvers be coordinated? As shown in figure 1, a problem can be partitioned along the constraints or along the variables. In the first case, the subproblems can be solved independently right from the start. But, when partial solutions are found, they must be tested against interface constraints. If some of these constraints are not satisfied, the resolution of the subproblems connected by these constraints has to be resumed. If the partition is made along the variables, we must start by looking for a consistent instantiation of the interface variables. After such an instantiation is found, each subproblem can be solved with complete independence as illustrated by figure 2. If they all succeed in finding a solution, we have a global solution. If no solution can be found for one of the subproblems, the instantiation of the interface variables should be changed and the resolution of the subproblems concerned by this change has to be resumed.

168 Let us suppose that the problem that we want to solve has n variables and is divided into k subproblems, each with p variables (so that n - kp). Each variable has a domain of d elements. Using a partition along the constraints, the worst case time complexity for finding the first solution is bounded by (dP)k which gives us a complexity of O(d~P). Now, using a partition along the variables and supposing that there are m interface variables, the worst case time complexity for finding the first solution is bounded by kdmd p which yields a complexity of O(dm+p). Of course, if the problem is not partitioned (i.e. k = 1, p = n and m = 0), we find the same usual complexity for both cases, i.e. O(dn). Conversely, when there are several subproblems, the theoretical complexity of the resolution with constraint-connected subproblems is much higher than with variable-connected subproblems. This is why we have chosen to investigate the latter mode of partition, keeping in mind that this choice is valid if and only if one solution is sufficient for our needs. 4. Definitions

Definition 1 (constraint p r o b l e m ) A binary constraint satisfaction problem P is pair of sets (V,C). The set ~; is the set of variables { v l , . . . , v n } . Each variable vi associated with a finite domain di where its values are sought. The set C is the set constraints. Each constraint is a pair of distinct variables {vi, vj} noted cij which associated with a relation rij that defines the set of allowed value pairs.

a is of is

Definition 2 ( s o l u t i o n ) A value assignment is a pair variable-value noted (v~, x~) where xi E di. A substitution a~ is a set of value assignments, one for each element in the set of variables E. A solution to the problem P = (~2, C) is a substitution av that satisfies all the constraints in C.

Definition 3 ( c o n s t r a i n t s u b p r o b l e m ) A subproblem P~ of P is a triple (Zi, ~)i,C~). The set Zi C V is the set of interface variables, ~)~ C )2 is the set of own variables, and Ci C C is the set of own constraints. Given an instantiation of its interface variables Zi, the solution to Pi is a substitution av~ that satisfies all the constraints in Ci. A subproblem has the following properties: P r o p e r t y 1 The sets of interface and own variables are disjoint i.e.: Zi N Vi = 0 P r o p e r t y 2 The set of own constraints is the maximal subset of the problem constraints that connect one own variable with either an own or an interface variable:

c~ = {C.b e Cl(vo, vb) 9 (V, x Z~) U (Z~ x V~) U (V~ x V~)}

Definition 4 (partition) A k-partition II~ of a problem P is a set of subproblems {P1,. 9 Pk} that have the following properties: k P r o p e r t y 3 The sets of own variables of the subproblems are disjoint i.e." N~=~ V~ = 0

169

Property 4 Each variable of the problem is either an interface or an own variable of a subproblem: k

k

(Ur~)u(UI~)=r ~=~ ~=1

k

and

k

(Uv~)n(Uz~)=O ~=~

~=~

Definition 5 (interface problem) The interface problem Pn of a partition I-I~ is a subproblem of P for which the own variable set is the union of the interface variable set of all the subproblems of the partition: k

))n = U Ii and I n = 0 i--1

Property 5 The set of constraints of the interface problem is the maximal subset of the problem constraints that connect any two of its own variables (that is any two interface variables from the other subproblems): c . = {c~b e Cl(v~ vb) e v . x v . }

T h e o r e m 1 Given a partition of P and a solution avn to its interface problem, the union of avn with any solution for all the subproblems of the partition constitutes a solution to the problem P. Proof." From the properties of a partition and the definition of the internee problem, k Gvi) instantiates once and only once each it is easy to show that the union avn U (Ui=I variable in 12 (from properties 1, 3 and 4) and that this union satisfies all the constraints k in 0 = On U (U~=~ 0~) (from properties 2 and 5). The union is therefore a solution to the whole problem P = (1;, 0). [] This theorem allows us to implement the resolution of a constraint problem as the resolution of an internee problem followed by an independent resolution of k subproblems. The following two sections describe shortly how to compute a problem partition and what coordination to implement between the parallel solvers.

5. P r o b l e m Partition Given k available processors and a problem P, the goal is to find a k-partition that best combines the following (possibly conflicting) desiderata: 1. The complexity of solving the different subproblems should be about the same. 2. The number of variables of the interface problem should be kept to a minimum. Of course, a complete exploration of the partition space is out of the question. We thus turn to a heuristics-based algorithm and we use the classic K-way graph partitioning algorithm presented in [15]. For our purposes, the cost of an edge is inversely proportional to the degree of satisfiability of the constraint represented by this edge. Therefore, a constraint that is easy to satisfy (i.e. with a high degree of satisfiability) has a low cost and will be preferred as a separating edge between two subproblems. The weight of a vertex is proportional to the domain size of the variable represented by this vertex. The set of interface variables is chosen as the minimal set of vertices that is connected to all the crossing edges determined by the partitioning algorithm.

170

1 while a new instantiation for interface variables can be found 2 3 4 5

instantiate all the interface variables for each subproblem Pi solve Pi as an independent CSP in case of success:

1,2

and 3}

store the partial solution {variants 2 and ,9}

6 7

{variants

in case of failure:

store the nogood {variants 2 and 3}

8 9

return to step 1 10 a solution is found; end. 11 no solution can be found; end.

Figure 3. Decomposition-based Search Algorithm Schema

6. A D e c o m p o s i t i o n B a s e d S e a r c h A l g o r i t h m The previous section was about how to get a good partition of the constraint problem. We have designed an algorithm for finding one solution that uses that problem partition to t.ry to solve each subproblem independently. The main idea of this algorithm is the following: first find a consistent instantiation for the interface variables in Vn, and solve each subproblem Pi with the so instantiated interface variables. As soon as a subproblem fails, we store the inconsistent instantiation (also known as a nogood) in order not to reproduce it and we look for a new consistent instantiation for the interface variables. The outline of the algorithm is presented in figure 3. From this outline, we can design three variants, depending on the backtracking schema for the interface variables and on the intermediate results we store. We use the following notations in the description of these three variants: 9 d: domain size 9 n: total number of variables 9 m: number of interface variables 9 p: maximum number of variables of a subproblem 9 s: maximum number of interface variables of a subproblem 9 k: number of subproblems 9 V a r i a n t 1: Standard backtracking on interface variables. The first idea is to use a standard chronological backtracking algorithm for the instantiation of the interface variables. No backtracking can occur between two subproblems and the reduction of worst case time complexity is then given by using

171 the constraint graph partition, as mentioned in section 3. For each instantiation of the interface variables, we have in the worst case one c o m p l e t e t r e e search for each subproblem, so the total complexity is in O(kdPd m) = O(kdp+m), instead of a complexity in O(d kp+m) for a global tree search algorithm. For that variant, the space complexity is the same as the global tree search algorithm: no additional storage is required. 9 V a r i a n t 2: Standard backtracking with storage of partial solutions and nogoods. In that second variant, we store the partial solutions and nogoods to ensure that each subproblem Pi will be solved only once for a given instantiation of its interface variables Z~. Then, the time complexity can be reduced to O(d TM + kdPd s) = O(d TM + kdP+S), the space complexity becoming O(kpdS), s being the maximum number of interface variables of one subproblem. 9 V a r i a n t 3: Dynamic backtracking on interface variables. We still store nogoods and partial solutions, as in the previous variant. We can notice that these nogoods can be used to implement a dynamic backtracking schema [16] for the instantiation of the interface variables. When a failure occurs in the solving of Pi, we try to change the instantiation of the interface variables Ii and to keep the instantiation of the other interface variables, which do not take part in the nogood. For that third variant, we have the same time and space complexity as for the preceding one. This variant is the decomposition-based search algorithm (DS) we have implemented. 7. Parallelization

7.1. A Parallel A l g o r i t h m with Centralized Control The DS algorithm we have presented can be naturally transformed into a parallel version PDS with k + 1 processes, one master for the instantiation of the interface variables and k slaves for solving of the subproblems. As for the sequential DS algorithm, given a partition II k, we first have to find a consistent instantiation as of the interface variables. This instantiation becomes the master's current instantiation. Once this instantiation is found, we can initiate the parallel processing of each subproblem. The master process, which is in charge of the interface instantiation, keeps waiting for the result of the slave processes. The result of the resolution of each subproblem can be either a failure or a success. In case of success, the partial solution of P~ is stored by the master in a dictiona~. (line 15). The slave in charge of Pi will then wait until the master restarts it further with another instantiation of its interface variables. In case of failure on a subproblem Pi, the subset ai of the current substitution as, corresponding to the interface variables of Pi is stored as a nogood (lines 6 and 15). The current substitution as of all interface variables is then invalid and it is necessa .ry to find a new combination of values for these variables. In order to not interrupt all the processes and let run some of them, the master will not follow a chronological backtracking, but a dynamic backtracking model, as seen in 6.2. Once this new instantiation a~ is found, we have to first interrupt, if they are running,

172

0 let Q be an empty message queue; 1 until exit from enumeration-end 2 instantiate consistently all the variables in ])n; 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17

Figure 4.

if no possible instantiation then exit from enumeration-end with no solution; else if failure in Q then store the nogood; else start in parallel the slaves processes whose interface variables have changed; tag reception-end while one of the slaves has not responded if Q is empty, then wait for a message; process the first message; in case of success: store the partial solution; in case of failure: store the nogood; exit from reception-end; exit from enumeration-end with solution;

PDS

algorithm for the master process

and then restart with the new instantiation of the interface variables the resolution of eve .ry subproblem Pj such that (lines 7 and 8): 3v 6 :Z'j,as(v) # ds(V ). Before restarting the resolution of a subproblem, the existence of a precomputed solution is looked-up in the dictiona .ry of partial solutions. The algorithm performed by the master process is presented in figure 4. 7.2. C o m m u n i c a t i o n s In this algorithm, the inter-process communications take place only between the master and the slaves:

9 communication from master to slaves The master is controlling the execution of the slaves. A message is an interruption that is immediately taken into account, as soon as the slave receives it. There is no message queue and the only possible message is: "start the resolution of the subproblem with the current instantiation of the interface variables." 9 communication from slave to master Here, a message corresponds to a report about the resolution of a subproblem. Either a partial solution was found, or a failure occurred. These messages are put in a queue, the master is not interrupted and it will process them in sequence. All the failures will be processed before restarting the slaves with another instantiation of their interface variables (lines 5, 6).

173

7.3. M e m o r y The storage of the nogoods and of the p~rtial solutions is done by the master. The slaves do not need to store any results. The memory requirement is the same as for the sequential DS algorithm (variant 3). 7.4. P r o c e s s Load B a l a n c i n g Due to synchronization, some processors can become idle. The master waits until one slave gives a result (line 11). A slave that gave an answer (a partial solution or a failure) will wait until the master relaunches it with another instantiation of its interface variables. If the new instantiation of the interface variables of a slave corresponds to an already solved subproblem, the processor remains idle. In future works, we will study a more efficient version of the coordination, where some idle processors can perform some search in advance, i.e. solve subproblems, with values of their interface variables different from the current values of these variables in the master. 8. E x p e r i m e n t a l E v a l u a t i o n

8.1. Tests on r a n d o m problems In order to study the performances of these decomposition-based algorithms, we have experimented with randomly generated constraint problems. The generation of random problems is based on the four usual parameters [17]: the number n of variables, the size d of the variables' domain, the constraint density cd in the graph and the constraint tightness ct. The constraint density corresponds to the fraction of the difference in the number of edges between a n-vertices clique and a n-vertices tree. A problem with density 0 will show n - 1 constraints; a problem with density 1 will show n(n - 1)/2 constraints. The constraint tightness ct corresponds to the fraction of the number of tuples in the cross-product of the domain of two variables that will not be allowed by the constraint between these two variables. Tightness 0 stands for the universal constraint and tightness 1, the unsatisfiable constraint. In our experiments, each problem we solved was made up of three constraint problems, generated with the same four parameters, and they were coupled by three interface variables, vl, v2, va, one for each subproblem. The coupling constraints were two difference constraints, vl ~- V2 and v2 r v3. We compared 3 algorithms: 1. A global algorithm which is the classical forward-checking with first fail (FC-FF), using a dynamic variable ordering, the smallest domain first. 2. The decomposition-based search algorithm, DS, which corresponds to variant 3 presented in section 6. 3. A simulation of the parallel version of the algorithm, PDS, presented in section 7. In order to have a fair comparison between these 3 algorithms, in the DS and PDS algorithms, the subproblems were solved with the same FC-FF algorithm used to solve the entire problem in the global algorithm. The parallelism of the third algorithm was simulated in one process, the communication time being then reduced to zero. One simulated processor was assigned to each subproblem.

174 8.2. E x p e r i m e n t a l Results We measure for each algorithm the cpu-time. In the case of the simulation of parallelism, the cpu-time we report is the execution time the parallel algorithm would have, if we suppose that the communications take no time. The results in figures 5, 6 and 7 report the time (in sec.) for solving sets of 40 problems, each set composed of subproblems generated with the same parameters. All tests reported here were run on problems made up of 3 subproblems composed of 25 variables with a domain size d equal to 7. In the figure 5, the constraint density cd is 0.2, and the constraint tightness ct is varying between 0.2 and 0.55. In the figure 6, the constraint density cd is 0.4, and the constraint tightness ct is varying between 0.1 and 0.4. In the figure 7, the constraint density cd is 0.6, and the constraint tightness ct is varying between .05 and 0.4. All these tests were run on a SUN Sparc 5 workstation, in a LELIsP implementation, the subproblems being solved with the PROSE [18] constraint toolbox. This toolbox offers some simple primitives to define constraints, build some constraint problems by connecting constraints and variables, and solve those problems. The main tool provided for resolution is parametrized tree search. The search can be adapted by selecting a variable ordering method, a value ordering method and a consistency method which is usually the forward-checking method. We have obtained three behaviors, depending on the difficulty of the problems. These behaviors correspond to under-constrained problems, to over-constrained problems and to problems in the phase transition zone [19,20]. U n d e r - C o n s t r a i n e d P r o b l e m s For the under-constrained problems (ct < 0.4 in figure 5, ct < 0.25 in figure 6, ct < 0.2 in figure 7), the global solution process is quite efficient, the DS algorithm doesn't improve the global algorithm very much. The parallelization is quite interesting in these cases: we have obtained between DS and P D S a speedup from 1.8 to 2.8. P h a s e Transition For the phase transition zone, (ct - 0.4 in figure 5, ct = 0.28 in figure 6, ct = 0.22 in figure 7), where some problems have few solutions and some are over-constrained, there exist problems that the global algorithm could not solve in 10 minutes. These problems, which were very difficult for the global algorithm, were solved by the decomposition-based algorithm in few seconds and the parallelization did not improve the results very much. We can explain that fact by the reduction of complexity thanks to the DS algorithm, which exploits the special structure of the constraint graph, while the standard forward checking algorithm with smallest domain heuristic cannot exploit it. O v e r - C o n s t r a i n e d P r o b l e m s For the over-constrained problems, (ct > 0.4 in figure 5, ct > 0.28 in figure 6, ct > 0.22 in figure 7), the uncoupling and the parallelization are not efficient. We can remark that the global forward checking with first fail algorithm focuses automatically on an unfeasible subproblem, and detecting that a subproblem is unfeasible is enough to deduce that the complete problem has no solution: in that case, the decomposition and the parallelization are useless. Furthermore, the decomposition-based algorithm changes the variable

175 200

|

|

i

|

w

!

run-time

i

"DS" -4--"PDS" --.~"FC-FF" -B--.

150

100

50 /jJ"

.15

.

i

i

!

i

i

0.2

0.25

0.3

0.35

0.4

0.45

0.5

0.55

Figure 5. Runtime comparison for 3 connected subproblems with n = 25, d = 7, cd = 0.2

ordering, beginning with the interface variables and this new ordering is often less efficient than the ordering given by the standard minimum domain heuristics.

8.3. Future Experiments These first results show that our decomposition-based algorithm outperforms the standard forward checking algorithm in the transition phase, and that the parallel version is interesting in the zone of under-constrained problems. Some other experiments should be done, varying the number of connections between the subproblems and the number of subproblems in order to confirm these results. We are now close to completing the implementation of our distributed solution process on a multi-computer architecture (i.e. a network of SUN Sparc 5 computers Connected by Ethernet and using the CHOOE protocol [21]). We will then be ready to apply our approach to some benchmark problems and evaluate correctly the cost of the communications through the network [22] and the possible workload imbalance. REFERENCES 1. E. Tsang. Foundations of Constraint Satisfaction. Academic Press, 1993. 2. B. Nudel. Consistent-labeling problems and their algorithms. Artificial Intelligence, 21:135-178, 1983. 3. A. Mackworth. Consistency in networks of relations. Artificial Intelligence, 8:99-118, 1977. 4. R. Haralick and G. Elliott. Increasing tree search efficiency for constraint satisfaction problems. Artificial Intelligence, 14:263-313, 1980. 5. D. Baldwin. CONSUL: A parallel constraint language. IEEE Software, 6(4):62-69, 1989.

176

,

,

,

,;

i :

.

!

,

"os"

/~

-.--

"PDS" ..4--.

8OO

5OO

3O0

100

0 0.05

0.1

0.15

0.2

0.25

0.3

0.35

0.4

Figure 6. Runtime comparison for 3 connected subproblems with n = 25, d = 7,

16001

'

'

i

'

i/~

i' i ~

'.. "pDDs S" ~-~ "FO-FF......

0.2

0.25

0.3

0.35

run-li rne

,~

I-

6

]~

~

cd

= 0.4

1200

1000

600

4O0

200

0

005

0.1

0.15

0.4

Figure 7. Runtime comparison for 3 connected subproblems with n = 25, d = 7,

cd --

0.6

177

10. 11. 12. 13. 14.

15. 16. 17. 18. 19. 20. 21. 22.

P. Cooper and M. Swain. Domain dependance in parallel constraint satisfaction. In Proc. IJCAI, pages 54-59, Detroit, Michigan, 1989. W. Hower. Constraint Satisfaction via Partially Parallel Propagation Steps, volume 590 of Lecture Notes in Artificial Intelligence, pages 234-242. Springer-Verlag, 1990. J. Conrad. Parallel Arc Consistency Algorithms for Pre-Processing Constraint Satisfaction Problems. PhD thesis, University of North Carolina, 1992. S. Kasif. On the parallel complexity of discrete relaxation in constraint satisfaction networks. Artificial Intelligence, 45:275-286, 1990. D. Warren. The SRI model for or-parallel execution of prolog. In International Symposium on Logic Programming, pages 92-101, 1987. R. Finkel and U. Manber. DIB: A distributed implementation of backtracking. A CM Transactions on Programming Language and Systems, 2(9):235-256, 1987. P. Van Henten .ryck. Parallel constraint satisfaction in logic programming. In Proc. ICLP, pages 165-180, 1989. Q. Y. Luo, P. G. Hendry, and J. T. Buchanan. A hybrid algorithm for distributed constraint satisfaction problems. In Proc EWPC'92, Barcelona, Spain, 1992. M. Yokoo, E. Durfee, T. Hishida, and K. Kuwabara. Distributed constraint satisfaction for formalizing distributed problem solving. In Proc of 12th IEEE International Conference on Distributed Computing Systems, pages 614-621, 1992. W. Kernighan and S. Lin. An efficient heuristic procedure for partitioning graphs. The Bell System Technical Journal, 49(1):291-307, 1970. M. Ginsberg. Dynamic backtracking. Journal of Artificial Intelligence Research, 1:25-46, 1993. D. Sabin and E. Freuder. Contradicting conventional wisdom in constraint satisfaction. In Proc. ECAI, pages 125-129, Amsterdam, Netherlands, 1994. P. Berlandier. PROSE : une boite A outils pour l'interpr~tation de contraintes : guide d'utilisation. Rapport Technique 145, INRIA Sophia Antipolis, 1992. P. Prosser. Binary constraint satisfaction problems : Some are harder than others. In Proc. ECAI, pages 95-99, Amsterdam, the Netherlands, 1994. B. Smith. Phase transition and the mushy region in constraint satisfaction problems. In Proc. ECAI, pages 100-104, Amsterdam, Netherlands, 1994. F. Lebastard. CHOOE: a distributed environment manager. Technical Report 93-22, CERMICS, Sophia-Antipolis (France), D~cembre 1993. P. Crandall and M. Quinn. Data partitioning for networked parallel processing. In Proc. 5th Symposium on Parallel and Distributed Processing, pages 376-380, Dallas, TX, 1993.

178 Pierre Berlandier Pierre Berlandier received his Ph.D. in computer science from INRIA and the University of Nice (France) in 1992. His research interests are focused on various aspects of constraint programming such as constraint satisfaction algorithms, consistency maintenance techniques and constraint-based languages design. He is now a senior consultant for ILOG Inc. in Mountain View, CA where he is working on constraint-based design and scheduling applications.

Bertrand Neveu

Bertrand Neveu graduated from Ecole Polytechnique and Ecole Nationale des Ponts et Chaussees. He worked as a research scientist at INRIA Sophia Antipolis since 1984 on the Smeci expert system shell project. He has then been leading the Secoia research project that focused on the design of AI tools using object oriented and constraint based knowledge representations. He is currently in charge of a constraint programming research team in the CERMICS laboratory in Sophia-Antipolis.

Parallel Processing for Artificial Intelligence 3 J. Geller, H. Kitano and C.B. Suttner (Editors) 1997 Elsevier Science B.V.

181

Parallel Propagation in the Description-Logic System FLEX* Frank W. Bergmann a and J. Joachim Quantz b aTechnische Universit~it Berlin, Projekt K I T - V M l l , FR 5-12, Franklinstr. 28/29, D-10587 Berlin, Germany, e-mail: [email protected] bTechnische Universit~it Berlin, Projekt K I T - V M l l , FR 5-12, Franklinstr. 28/29, D-10587 Berlin, Germany, e-mail: [email protected] In this paper we describe a parallel implementation of object-level propagation in the Description-Logic (DL) system FLEX. We begin by analyzing the parallel potential of the main DL inference algorithms normalization, subsumption checking, classification, and object-level propagation. Instead of relying on a parallelism inherent in logic programming languages, we propose to exploit the application-specific potentials of DLs and to use a more data-oriented parallelization strategy that is also applicable to imperative programming languages. Propagation turns out to be the most promising inference component for such a parallelization. We present two alternative PROLOG implementations of paralle!ized propagation on a loosely coupled MIMD (Multiple Instruction, Multiple Data) system, one based on a .farm strategy, the other based on distributed objects. Evaluation based on benchmarks containing artificial examples shows that the farm strategy yields only poor results. The implementation based on distributed objects, on the other hand, achieves a considerable speed-up, in particular for large-size applications. We finally discuss the impact of these results for real applications. 1. I N T R O D U C T I O N In the last 15 years Description Logics (DL) have become one of the major paradigms in Knowledge Representation. Combining ideas from Semantic Networks and Frames with the formal rigor of First Order Logic, research in DL has focussed on theoretical foundations [1] as well as on system development [2] and application in real-world scenarios

[3-5]. Whereas in the beginning it was hoped that DL provide representation formalisms which allowed efficient computation, at least three trends in recent years caused efficiency problems for DL systems and applications: 9 a trend towards expressive dialects; *This work was funded by the German Federal Ministry of Education, Science, Research and Technology (BMBF) in the framework of the Verbmobil Project under Grant 01 IV 101 Q 8. The responsibility for the contents of this study lies with the authors.

182 9 a trend towards complete inference algorithms; 9 a trend towards large-scale applications. With the current state of technology it seems not possible to build a DL system for largescale applications which offers an expressive dialect with complete inference algorithms. The standard strategy to cope with this dilemma is to restrict either expressivity, or completeness, or application size. In this paper we investigate an alternative approach, namely a parallelization of Description Logics. Due to physical limitations in performance gains in conventional processor architectures, parallelization has become more and more important in recent years. This comprises parallel structures inside processors as well as outside by scaling several processors to parallel systems. Several fields of high-performance computing already adopted to this new world of paradigms, such as image processing [6], finite element simulation [7], and fluid dynamics [8]. We expect that in the future parallelism will become a standard technique in the construction of complex AI applications. A standard approach to parallelization in the context of logic programming concentrates on the development of parallel languages that exploit the parallelism inherent in the underlying logic formalism ([9,10] and many more). In this paper we will follow a rather different approach which analyzes a particular application, namely Description Logics. The parallelization we propose uses explicit parallelism based on the notion of processes and messages that is programming language independent. In the next section we give a brief introduction into Description Logics. Section 3 then pi'esents the main inference components of the DL system FLEX and investigates their parallel potential. In Section 4 we describe two different strategies of parallelizing objectlevel propagation in DL systems. The corresponding implementations are evaluated in detail in Section 5 based ' On benchmarks containing artificial examples. Section 6 finally discusses the impact of these results on real applications. 2. D E S C R I P T I O N

LOGICS

In this section we give a brief introduction into Description Logics. Our main goal is to provide a rough overview over DL-based knowledge representation and DL systems. In the next section we will then take a closer look at inferences in the DL system FLEX and their respective parallel potential. 2.1. T h e R e p r e s e n t a t i o n L a n g u a g e In DL one typically distinguishes between terms and objects as basic language entities from which three kinds of formulae can be formed: definitions, descriptions, and rules (see the sample model on page 3 below). A definition has the form 'tn:= t' and expresses the fact that the name tn is used as an abbreviation for the term t. A list of such definitions is often called terminology (hence also the name Terminological Logics). If only necessary but no sufficient conditions of terms are specified a definition has the form 'tn:< t', meaning that 'tn' is more specific than 't'. Terms introduced via ' : - ' are called defined terms, those introduced via ':<' are primitive terms.

183 All DL dialects provide two types of terms, namely concepts (una .ry predicates) and roles (binary predicates), but they differ with respectto the term-forming operators they contain. Common concept-forming operators are: conjunction (cl and c2), disjunction (cl or c2), and negation (not(c)), as well as quantified restrictions such as value restrictions (all(r,c)), which stipulate that all fillers for a role r must be of type c, or number restrictions (atleast(n,r,c), atmost(n,r,c)), stipulating that there are at least or at most n role-fillers of type c for r. Role-forming operators are, besides conjunction, disjunction, and negation, role composition (rl comp r2), transitive closure (trans(r)), inverse roles (inv(r)) and domain or range restrictions (domain(c), range(r,c)). In a description, an object is described as being an instance of a concept (o :: c), or as being related to another object by a role (ol :: r:o2). Rules have the form ' c 1 - > c2' and stipulate that each instance of the concept cl is also an instance of the concept c2. In general, the representation language is defined by giving a formal syntax and semantics. Note that DL are subsets of First-Order Logic (with Equality), which can be shown easily by specifying translation functions from DL formulae into FOL formulae [11,12]. Just as in FOL there is thus an entailment relation between (sets of) DL formulae, i.e. a DL model can be regarded as a set of formulae F which entails other DL formulae (F ~ 7)- Depending on the term-forming operators used in a DL dialect this entailment relation can be decidable or undecidable and the inference algorithms implemented in a DL system can be complete or incomplete with respect to the entailment relation. 2.2. A S a m p l e M o d e l In order to get a better understanding of DL-based knowledge representation let us take a look at the modeling scenario assumed for applications. An application in DL is basically a domain model, i.e. a list of definitions, rules, and descriptions. Note that from a system perspective a model or knowledge base is thus a list of tells, from a theoretical perspective it is a set of DL-formulae F. Consider the highly simplified domain model below, whose net representation is shown in Figure 1. One role and five concepts are defined, out of which four are primitive (only necessa .ry, but no sufficient conditions are given). Furthermore, the model contains one rule and four object descriptions. product chemical product biological product company produces chemical company some(produces,chemical product) toxipharm biograin chemoplant toxiplant

:< :< :< :< :< := => :: :: :: ::

anything product product & not(chemical product) some(produces,product) domain(company) company & all(produces,chemical product) high risk company chemical product biological product chemical company atmost(1,produces) & produces:toxipharm

As mentioned above such a model can be regarded as a set of formulae and the service provided by DL systems basically is to answer queries concerning the formulae entailed

184

1..in

company

O

produces

high risk

~

~

9 fr

chemical ~ ( product j , . r _ . . ~ ~

biological product

chemical company

chemoplant

toxiplant

toxipharm

biogmin

produces

Figure 1. The net representation of the sample domain on page 3. Concepts are drawn as ellipses and are arranged in a subsumption hierarchy. Objects are listed below the most specific concepts they instantiate. Roles are drawn as labeled horizontal arrows (annotated with number and value restrictions) relating concepts or instances. The dashed arrow relates the left-hand side of a rule with its right-hand side ('conc_l' is the concept 'some(produces,chemical product)'). The flashed arrow between 'chemical product' and 'biological product' indicates disjointness.

185 by such a model. The following list contains examples for the types of queries that can be answered by a DL system: 9

tl ?< t2 Is a term tl more specific than a term t2, i.e., is tl subsumed by t27 In the sample model, the concept 'chemical company' is subsumed by 'high risk company', i.e., eve .ry chemical company is a high risk company. 2

9 t l and t2 ?< nothing Are two terms tl and t2 incompatible or disjoint? In the sample model, the concepts 'chemical product' and 'biological product' are disjoint, i.e., no object can be both a chemical and a biological product. .o?:c Is an object o an instance of concept c (object classification)? In the sample model, 'toxiplant' is recognized as a 'chemical company'. 9 o17:r:o2

Are two objects ol,o2 related by a role r, i.e., is 02 a role-filler for r at o1? In the sample model, 'toxipharm' is a role-filler for the role 'produces' at 'toxiplant'. 9 Which objects are instances of a concept c (retrieval)? In the sample model, 'chemoplant' and 'toxiplant' are retrieved as instances of the concept 'high risk company'. 9 o::cfails

Is a description inconsistent with the model (consistency check)? The description 'chemoplant :: produces:biograin' is inconsistent, with respect to the sample model, i.e., 'biograin' cannot be produced by 'chemoplant'. 3 This very general scenario can be refined by considering generic application tasks such as information retrieval, diagnosis, or configuration. 2.3. S y s t e m I m p l e m e n t a t i o n s From the beginning on, research in DL was praxis-oriented in the sense that the development of DL systems and their use in applications was one of the primary interests. In the first half of the 1980's several systems were developed that might be called in retrospection first-generation DL systems. These systems include KL-ONE, NIKL, KANDOR, KL-TWO, KRYPTON, MESON, and SB-ONE. In the second half of the 1980's three systems were developed which are still in use, namely BACK, CLASSIC, and LOOM. The LOOM system [13] is being developed at USC/ISI and focuses on the integration of a variety of programming paradigms aiming at a general purpose knowledge representation system. CLASSIC [2] is an ongoing 2'chemical company' is defined as a 'company' all whose products are chemical products; each 'company' produces some 'product'; thus 'chemical company' is subsumed by 'some(produces,chemical product)' and due to the rule by 'high risk company'. 3Object tells leading to inconsistencies are rejected by DL systems.

186 AT&T development. Favoring limited expressiveness for the central component, it is attempted to keep the system compact and simple so that it potentially fits into a larger, more expressive system. The final goal is the development of a deductive, object-oriented database manager. BACK [14] is intended to serve as the kernel representation system of AIMS (Advanced Information Management System), in which tools for semantic modeling, defining schemata, manipulating data, and que .rying, will be replaced by a single high-level description interface. To avoid a "tool-box-like" approach, all interaction with the information reposito .ry occurs through a uniform knowledge representation system, namely BACK, which thus acts as a mediating layer between the domain-oriented description level and the persistency level. The cited systems share the notion of DL knowledge representation as being the appropriate basis for expressive and efficient information systems [15]. In contrast to the systems of the first generation, these second generation DL systems are full-fledged systems developed in long-term projects and used in various applications. The systems of the second generation take an explicit stance to the problem that determination of subsumption is at least NP-hard or even undecidable for sufficiently expressive languages: CLASSIC offers a ve.ry restricted DL and almost complete inference algorithms, whereas LOOM provides a ve~. expressive language but is incomplete in many respects. Recently, the KRIS system has been developed, which uses tableaux-based algorithms and provides complete algorithms for a ve.ry expressive DL [16]. KRIS might thus be the first representative of a third generation of DL systems, though there are not yet enough experiences with realistic applications to judge the adequacy of this new approach. 4 In the following section we describe the FLEX System, which can be seen as an extension of the BACK system. FLEX is developed at the Technische Universit~it Berlin within the project KIT-VM11, which is part of the German Machine-Translation project VERBMOBIL. 3. T H E F L E X S Y S T E M Having sketched some of the general characteristics of DL we will now turn our attention towards a specific DL system, namely the FLEX system [18]. Compared to its predecessor, the BACK System, FLEX offers some additional expressivity such as disjunction and negation, term-valued features, situated descriptions, and weighted defaults. In the context of this paper another characteristic of FLEX is more important, however, namely the one giving rise to its name, i.e. flexible inference strategies. 3.1. D L I n f e r e n c e s Given the specification of DL in the previous Section, the inference algorithms have to answer queries of the form o tl

?: ?<

c t2

with respect to a knowledge base containing formulae of the form 4The missing constructiveness of the refutation-oriented tableaux algorithms (see next section) leads to problems with respect to object recognition and retrieval (see [17]).

187 tn tn

:---:(

El

=~

0

::

t t C2 C

Two things are important to note: 1. The FLEX system already performs inferences when reading in a model. There are two major inference components, namely the classifier and the recognizer. The classifier checks subsumption between the terms defined in the terminology and thus computes the subsumption hierarchy. The recognizer determines for each object which concepts it instantiates and thus computes the instantiation index. 2. For answering both kinds of queries, the same method can be used, namely subsumption checking. Thus when answering a query 'o ?: c', the system checks whether the normal form derived for 'o' is subsumed by 'c'. Though the recognizer thus uses the classifier to perform its task, there is an important difference between the two components. Whereas the classifier performs only "local" operations, the recognizer has to perform "global" operations. This distinction can be illustrated by briefly sketching the algorithmic structure of both components. Given a list of definitions, the classifier takes each definition and compares it with all previously processed definitions, thereby constructing a directed acyclic graph called the subsumption hierarchy. Thus the concept classifier is a function 'Concept x DAG --+ DAG', where the initial DAG contains the nodes 'anything' and 'nothing'. Locality here means that classifying a concept has no impact on previous classification results, i.e. classifying concept Ca has no impact on the subsumption relation between cl and c2. Recognition, on the other hand, has global effects. Basically, the recognizer processes a list of descriptions and computes for each object which concepts it instantiates, i.e. it is a function 'Description x Index x DAG--+ Index'. Nonlocality here means that recognition triggered by a description 'ol :: c' can lead to changes in the instantiation index for some other object 02, as exemplified by Ol Ol

:: ::

r:o2 all(r,c)

Here processing the second description includes the derivation of '02 :: c' Note that another distinction between classification and recognition is thus that there is exactly one definition for each term in the terminology, whereas objects can be described incrementally, i.e. we can have several descriptions for an object in a situation. In the following we will briefly describe the algorithmic structure of normalization, subsumption checking, and object-level propagation. Before doing so, however, we will first present the proof-theoretical basis of these algorithms.

3.2. Tableaux-Based Algorithms vs. Normalize-Compare Algorithms The first classifiers for DL were specified as structural subsumption algorithms [11]. The basic idea underlying structural subsumption is to transform terms into canonical normal forms, which are then structurally compared. Structural subsumption algorithms

188 are therefore also referred to as normalize-compare algorithms. Note that there is a general tradeoff between normalization and comparison: the more inferences are drawn in normalization, the less inferences have to be drawn in comparison, and vice versa. There is one severe drawback of normalize-compare algorithms--though it is in general straightforward to prove the correctness of such algorithms there is no method for proving their completeness. In fact, most normalize-compare algorithms are incomplete which is usually demonstrated by giving examples for subsumption relations which are not detected by the algorithm [19]. At the end of the 1980's tableaux methods, as known from FOL (cf. [20, p. 180fl]), were applied to DL (e.g. [1,21]). The resulting subsumption algorithms had the advantage of providing an excellent basis for theoretical investigations. Not only was their correctness and completeness easy to prove, they also allowed a systematic study of the decidability and the tractability of different DL dialects. The main disadvantage of tableaux-based subsumption algorithms is that they are not constructive but rather employ refutation techniques. Thus in order to prove the subsumption 'cl :< c2' it is proven that the term 'cl and not(c2)' is inconsistent, i.e. that 'o :: Cl and not(c2)' is not satisfiable. Though this is straightforward for computing subsumption, this approach leads to efficiency problems in the context of retrieval. In order to retrieve the instances of a concept 'c', we would in principle have to check for each object 'o' whether F U {o :: not(c)} is satisfiable, where F is the knowledge base. 5 In most existing systems, on the other hand, inference rules are more seen as production rules, which are used to pre-compute part of the consequences of the initial information. This corresponds more closely to Natural Deduction or Sequent Calculi, two deduction systems also developed in the context of FOL. A third alternative, combining advantages of the normalize-compare approach and tableaux-based methods has therefore been proposed in [12]. The basic idea is to use Sequent Calculi (SC) instead of tableaux-based methods for the characterization of the deduction rules. Like tableaux methods, Sequent Calculi provide a sound logical framework, but whereas tableaux-based methods are refutation based, i.e. suitable for theorem checking, sequent calculi are constructive, i.e. suitable for theorem proving. By rewriting SC proofs of FOL translations of DL formulae we obtain sequent-style DL inference rules like rl :< r2, cl :< c2 ~ cl and c2:< nothing, h : < r2, p > 0 ~

all(r2,cl) :< all(rl,c2) all(r2,c2) and atleast(p,rl,cl) :< nothing

The first rule captures the monotonicity properties of the all operator [23, 24]. If 'has_daughter' is subsumed by 'has_child' and 'computer_scientist' is subsumed by 'scientist', then 'all(has_child,computer_scientist)' is more specific than 'all (has_daughter,scientist)'. The second rule means that a qualified minimum restriction combined with a value restriction which is disjoint from the concept used in the minimum restriction is incoherent. If 'european' and 'american' are disjoint concepts and 'workshop_participant' 5See [22] for tableaux-based algorithms for object-level reasoning and [17] for a discussion of efficiency problems.

189 subsumes 'workshop_speaker', then the concept 'all(workshop_participant,american) and some(workshop_speaker,european)' is incoherent, i.e. is subsumed by 'nothing'. Note that this format is sufficient for a theoretical characterization of a deduction system, i.e. given a set of inference schemata E we can define a least fixed point r by taking the logical closure of a set of formulae F under E. We can then say that F k-z "7 iff e r ~ Though we can study formal properties like soundness or completeness, i.e. the relation between F k-z -y and F ~ % on the basis of this characterization, we need an additional control strategy for turning the deduction system into an algorithm. The main reason for this is that r is not finite. The sequent-style approach thus falls into two separate phases: 1. Derivation of a complete axiomatization by systematically rewriting FOL proofs. 2. Specification of a control strategy to turn the complete axiomatization into an algorithm. In the second phase we have to turn the sequent-style inference rules into normalization rules and subsumption rules. As it turns out, some inference rules can be straightforwardly encoded as both normalization and subsumption rules, while others can only be represented as normalization or subsumption rules, respectively. The idea of a flexible inference strategy then means that each inference rule can be used during normalization, during subsumption or switched off completely. Note that this "customization" of the inference engine of FLEX should be performed by the system developers according to the inference requirements arising in a particular application. 3.3. N o r m a l i z a t i o n For each object, concept, and role, the FLEX system computes a normal form. The basic building blocks of normal forms are so-called atoms, which correspond to the termbuilding operators of the DL, e.g. 'all(R,C)', 'fills(R,O)'. Note that the R's and C's occurring in the atoms are themselves normal forms of roles and concepts. One way of formally defining atoms and normal forms is thus by means of a parallel inductive definition, as done in [12]. For languages not containing negation or disjunction, a normal form is simply a set of atoms. Since the FLEX system supports both disjunction and negation, however, we use the format of disjunctive normal forms proposed in [26]. In the context of this paper, however, it is sufficient to consider the simpler case in which normal forms are represented as sets of atoms. When reading in a concept or a role the parser already performs certain kinds of normalization, such as eliminating negation. Thus given a concept definition 'cn:= c' or an object description 'o :: c', the system transforms the term 'c' into a normal form. This normal form is then further processed by applying normalization rules. The basic idea of normalization is to make information implicitly contained in a normal form explicit, i.e. NORMALIZE: NF -~ NF 6See [25] for a formal definition.

190 The general format of a normalization rule is: 7 0/1,

.

.

.

,

Ot n

~

O~

The application of such a normalization rule is triggered when a normal form contains the atoms a l , . . . , an, in which case a is added to the normal form. The parallel potential of normalization consists in the possibility to apply all normalization rules in parallel. However, the results of applying normalization rules then have to be synchronized in a second step. 3.4. S u b s u m p t i o n C h e c k i n g The task of subsumption checking is to decide whether a normal form subsumes another normal form, i.e. SUBSUMES: NF x NF -~ BOOL As has been mentioned above, there is a general trade-off between normalization and subsumption checking. The more inferences are performed during normalization, the less inferences have to be drawn during subsumption checking. In principle it is possible to produce normal forms which guarantee that subsumption checking only has to test subsumption between individual atoms. Such vivid normal forms have been presented for a simple fragment of DL in [12, Sect. 5]. The disadvantage of such vivid normal forms is that they require the application of many normalization rules, making information explicit which might never be needed. Performing inferences during subsumption checking on the other hand guarantees that inferences are only drawn when actually needed. 8 Basically, subsumption checking between normal forms is ultimately reduced to subsumption checks between atoms, but includes also special subsumption rules for disjunctive parts, non-disjunctive parts, etc. This reduction of subsumption offers the possibility to perform subsumption tests between the atoms in the normal forms in parallel. It should be noted, however, that these atomic subsumption tests are rather fine-grained and unevenly distributed tasks. 3.5. Classification As already sketched above, the primary task of the classifier is to compute a subsumption hierarchy for the terms defined in the terminology. Thus given the normal form of a term, the classifier has to find its direct subsumers and its direct subsumees in the hierarchy. This can obviously achieved by comparing the normal form with all normal forms contained in the hierarchy, i.e. by checking the respective subsumption relations. The number of subsumption checks can be reduced, however, by taking into account the information already contained in the hierarchy. Thus if 'tl' subsumes 't2' and the subsumption test between the normal form and 't2' fails, there is no need to test subsumption between the normal form and 'tl'. Such optimization techniques are employed in all DL systems and are discussed in some detail in [16]. TThis is a further simplification since there are more complicated normalization rules involving disjunctions of atoms. Sin general, a mixed strategy is needed, which ensures both efficient performance and detection of inconsistency already during normalization.

191 The FLEX system uses different forms of classification which are based on a single algorithm, however. When processing an object description, for example, the object's normal form is first classified only with respect to a subset of the subsumption hierarchy, namely the left-hand sides of rules. Moreover, this classification step only computes direct subsumers of the object's normal form, since there is no need to determine the subsumees of objects. The parallel potential of classification obviously is given by the possibility to perform the comparisons of a normal form with all the normal forms contained in the hierarchy in parallel. However the more efficient algorithms utilize the fact that subsumption tests within a classification task are not independent from each other. Consequently, algorithms using this fact pose an order on the subsumption tests and thus loose the property of being straightforwardly parallelizable. 3.6. O b j e c t - L e v e l R e a s o n i n g

As already indicated above, object-level reasoning is inherently non-local and it is therefore useful to distinguish between a local phase and a non-local phase in object-level reasoning. In the local phase we determine for an object the most specific concept it instantiates. This can be done by using the standard normalize and compare predicates. Thus we normalize the description of an object thereby obtaining a normal form and compare it with the normal forms of the concepts in the hierarchy. In addition to this standard classification we also have to apply rules when processing objects. This is achieved by applying all rules whose left-hand sides subsume the object's normal form. After this application the normal form is again normalized and classified until no new rules are applicable [27]. In the non-local phase we have to propagate information to other objects. There are five basic rules for propagation, whose effects we will illustrate below with examples:

01 ::.all(r,c), 01 :: r:02 -~

o2 :: c

01 :: all(r,oneof([ol,...,on])), 02 :: c,...,on :: c --~ 01 :: all(r,c) 01 :: r:02, o2 :: c -~

01 :: (r and range(c)):02

01 :: h:02, o2 :: r2:03 -~ o2 :: (rlcomp r2):03 01 :: r:02 - -

o2:: inv(r):01

(1) (2) (3) (4) (5)

Rule (1) is usually called forward propagation since the value restriction is propagated to each individual filler of a role. Rule (2) is called backward propagation or abstraction over closed role-filler sets--if all fillers of a role are known the most specific value restriction for that role can be abstracted. The other three rules are related to role-forming operators. Rule (3) says that a filler for a role 'r' which is an instance of a concept 'c' is automatically a filler for the role 'r and range(c)'. Rules (4) and (5) capture the semantics of role composition and inversion, respectively. The following examples illustrate the effects of these propagation rules:

192

~(5) (3) (2)

(4) (1)

has_child has_parent has_daughter has_grandchild father happy_father john john mary mary john mary john mary mary john ma.ry

:< := := := := := :: :: :: :: :: :: :: :: :: :: ::

domain(human) and range(human) inv(has,child) has_child and range(female) has_child comp has_child male and some(has_child) father and all(has_daughter,married) male and has_child:ma.ry and exactly(1,has_child) father has_parent:john female has_daughter:mary married happy_father has_daughter:kim has_child:kim has_grandchild:kim female

Note that this sequent-style characterization of object-level reasoning is sufficient for a theoretical characterization of a deduction system (see Section 3.2). The actual propagation algorithm of the FLEX system consists of these rules together with data structures and predicates which trigger the appropriate rules. Basically, whenever an object is processed, we collect propagations of the form 'O :: c' resulting from the five propagation rules. First, it should be noted that object-level propagation is a rather coarse-grained task, since adding information to an object involves normalization and classification (and hence subsumption checking). Furthermore, processing propagations in parallel does not require any additional synchronization of propagation results. Processing a propagation can trigger additional propagations, but these can be processed by the same parallel mechanism. Finally, object-level propagation poses the main performance problem in most DL applications. In Section 6 we sketch some mechanisms to reduce the complexity of objectlevel reasoning in the sequential FLEX system and compare their effects to the speed-up obtained by our parallel implementation. 3.7. P a r a l l e l P o t e n t i a l of Inference A l g o r i t h m s in F L E X To summarize this section on the FLEX system we briefly recapitulate the results concerning the parallelization potential of the different inference algorithms. Figure 2 shows which inference components are used by which other inference components. First, we note that normalization, subsumption checking and classification share the following negative properties with respect to parallelization: 1. they are rather fine-grained tasks; 2. the distribution of the task length is rather widespread; 3. they involve a synchronization step that is not required in the parallelization of object-level propagation.

193 Object-Level Propagation

Classification

~

USeS

Normalization

Subsumption Checking

Figure 2. Dependencies between the DL Inference components.

Object-level propagation, on the other hand, has the following positive characteristics with respect to parallelization: 1. propagation involves coarse-grained tasks; 2. many applications involve a high number of propagations; 3. propagations are independent from each other, monotonic, and invariant with respect to the order of execution; 4. synchronization is only required before an object query ist stated (to make sure that all propagations have terminated); 5. the propagation algorithm tion/communication ratio.

can

be

implemented

with

a

high

computa-

We therefore concentrate on the parallelization of object-level propagation in the following. 4. P A R A L L E L I Z A T I O N S T R A T E G I E S In the remainder of this paper we investigate two parallel implementations of objectlevel propagation. The following properties of propagation should be kept in mind: 1. the length of propagation tasks is not known a priori; 2. the number of propagation tasks is not known a priori; 3. the "direction" of propagation flow is not known a priori. We will briefly illustrate these properties by analyzing the propagation data flow.

194

initial propagation

Figure 3. A group of objects interchanging propagations.

Hems

I000I

/

~

i00

Pending Propagations

/

N

k Iz

I0 Propagation o

2

4

6

8

1o

12

14

16

18

I

l

Figure 4. Exponential increase of propagations.

20

Steps

195

lime

Figure 5. Timing of the farm communication scheme.

4.1. F L E X D a t a F l o w We begin by noting several relevant properties of object-level propagation. As already indicated above propagation of information from one object to another can cause additional propagation to other objects. This kind of 'ping-pong' interaction terminates only when a 'fixed point' is reached and no new information is produced. Since propagation in Description Logics is monotonic, we can execute propagations in an arbitra .ry order, always ending up with the same result. We will refer to this property as confluence. For illustration consider the artificial example in Figure 3. The initial propagation affects object '02' and processing this object yields propagations to objects 'o1', '06', and 'o8', etc. The propagation at '02' thus creates a "chain reaction", i.e. the number of pending propagations increases exponentially. After some time the new information is integrated into the network and the pending propagations decrease until the fixed point is reached. For a parallel processing of propagations we thus obtain the work distribution qualitatively sketched in Figure 4. In the initial phase there are only few propagations and hence idle processors, in the middle phase there are more pending propagations than processors, and in the final phase there are again only few propagations and hence idle processors. Thus the propagation steps in the middle part will take longer to process since they cannot be completely distributed to the limited number of processors available. Given the analysis of the FLEX data flow, we consider two parallel paradigms as potential candidates for an implementation: The farm paradigm and the distributed objects paradigm. In the remainder of this section we briefly present these two alternatives. Theoretical considerations and numerical calculations towards efficiency and scalability can be found in the detailed analysis in [28].

196

tim

Figure 6. Communication events and workload distribution during the first two propagation stages.

4.2. Farm Parallelism The .farm communication structure shown in Figure 5 is widely used in industrial applications such as image processing [6] and finite element simulation [7]. It is theoretically well known and there exists a variety of strategies to distribute workload evenly across a network. A farm consists of several parallel processes with a fixed structure: one process is called 'master' and is responsible to distribute tasks to a group of 'worker' processes which perform their tasks in parallel and return control and results back to the master. Farm structures are frequently used to parallelize applications that can be split into subtasks with a priori known duration. Examples are image processing or finite element systems. From a theoretical point of view, there are two potential sources of inefficiency in this architecture: 1. uneven distribution of workload and 2. a communication bottleneck created by the centralized position of the master.

4.3. C o m m u n i c a t i n g Objects Parallelism In the communicating-objectsparadigm the central management institution (master) of the farm parallelism is replaced by (local) knowledge of all objects about the 'addresses' of

197

,s+r.o,,..~..~ . ~ . .

_

:

-_

. . - - _ - _ - - - . -

-

-.

gN, g,,l : :,

Topology Processors Workers

2xl 3 1

3xl 4 2

2x2 5 3

", -:%

3x2 7 5

3x3 10 8

4x3 13 11

4x4 17 15

Figure 7. Hardware configuration and working nodes.

their neighbors. Objects communicate directly with each other, in contrast to the centered communication scheme of the farm. This helps to avoid communication bottlenecks in a network. The general difference between a farm and a network of communicating objects is the different perspective of parallelism: Within a farm, tasks are distributed; within the distributed objects scheme, objects are distributed. This approach appears to be similar to the agent-based paradigm developed by distributed AI research [29]. In contrast to this approach, objects within FLEX have to be considered elements of a distribution strategy rather then independently interacting entities. With respect to the definition given in [30] we have to subsume our efforts here under the field of 'distributed problem solving'. For an effective balancing of workload, certain assumptions about tasks and the computational environment have to be made. In our case, all processors can be assumed to behave identical and the statistical distribution of the task length is assumed to be narrow. Uneven distributions of workload can finally be treatedby special load balancing algorithms (see below).

5. E X P E R I M E N T A L

RESULTS

In this section we describe the hardware and the software used for implementing and evaluating our parallel implementations of FLEX. We also present benchmark results for artificial examples. In the next section we discuss the connection between these benchmark evaluations and real applications.

198 cl c2 c3

:< :< :<

all(r,c2) all(r,c3) all(r,cl)

ol o2 03 04 05 06 07 08 ol

:: :: :: :: :: :: :: :: ::

r:o3 r:o4 r:o7 r:ol r:o2 r:ol r:o3 r:o7 cl

and and and and and and and and

r:o2 r:o7 r:o2 r:o8 r:o7 r:o7 r:o8 r:o4

and and and and and and and and

r:o8 r:o2 r:ol r:o6 r:o8 r:o5 r:o4 r:o6

Figure 8. A sample benchmark with 8 objects, 3 concepts and fan-out 3.

5.1. H a r d w a r e and Software We chose the 'Parsytec Multicluster II' machine as base for the parallel implementation of FLEX. It consists of 16 processing nodes that each contain an INMOS T800 Transputer with 4 MByte of RAM. Each Transputer consists of a RISC processing kernel, a memo .ry interface and 4 DMA driven serial interfaces, called 'links'. Each link has a transfer rate of approximately 1.2 MByte/s and all 4 links can run independently together with the RISK kernel, hardly: affecting processing performance (communication delays could be neglected). This hardware platform is especially suitable to serve as a testbed for parallel implementations due to its simple architecture and the availability of comfortable development environments. However, it does not provide state of the art computational performance and suffers substantially from memo .ry restrictions. Figure 7 shows the topologies used for the tests in this section and the number of available worker nodes. The overhead of 2 processors is due to memo .ry limitations. Processor I could not be used because its 1 MByte RAM is not sufficient to hold the FLEX code. Processor 2 is used to hold the 'shell' process that synchronizes the generation of new objects. Normally this process can be located somewhere in the network and would not consume any computing performance, but in this case it had to be separated due to memo .ry restrictions. The language used to implement Distributed FLEX is a Prolog dialect called Brain Aid Prolog (BAP)[31]. It represents a standard Prolog with parallel libra .ry extensions, implementing a scheme similar to CSP [32]. Parallelism and synchronization is expressed explicitly using the notion of processes and messages. A process in BAP is a single and independent Prolog instance with a private database. A message is any Prolog term that is exchanged between two processes. Messages are sent and received using the 'send_msg(Dest, Msg)' and 'rec_msg(Sender, Msg)' predicates. Message sender and destination are identified by their process id' (PID). Messages are routed transparently through the network. The order of messages is maintained when several messages are sent from the same sender to the same destination. When a message reaches its destination process, it is stored in a special database, called 'mailbox'. Each process owns its private

199

el 0_3_2 c20_3_2

(seq) (58) (177)

1 78 253

2

3

5

63 185

55 159

56 160

8 58 162

Figure 9. Execution times (seconds) for the farm parallelization.

mailbox in which the messages are stored FIFO. Although the development of parallel FLEX was greatly simplified by the way BAP expresses parallelism, it is possible to apply the same parallel techniques to future FLEX implementation in programming languages such as LISP or C. 5.2. B e n c h m a r k s

To evaluate our parallel implementations we developed a benchmark generator that is capable of generating randomly connected networks of objects. The basic structure of these benchmarks is shown in Figure 8, which contains a benchmark consisting of 8 objects, 3 concepts, and a fan-out of 3. Thus, the final object tell 'ol :: cl' triggers propagation to three objects, namely 'o2', '03', and 'o8'. Each of these propagations again triggers 3 more propagations, etc. It should be obvious that the average fan-out, i.e. the number of additional propagations triggered by each propagation is a measure to describe the "avalanche" effect and is the major factor for the system's performance. 9 Figure 9 shows the execution times for the farm benchmarks. The first row contains the benchmark names that are composed by three numbers that indicate the number of objects, concepts and fan-out respectively (for example 'c20_5_3' means that the benchmark consists of 20 objects, 5 concepts and a fan-out of 3). The following rows give the execution times with respect to the number of processors. The '(seq)' fields gives the reference time of the (original) sequential version of FLEX. The parallelization of FLEX using the farm paradigm showed very poor results. This can be explained by the rather high costs to distribute the system state out to the workers and to integrate the computation results back into the system state. Both activities have to be done sequentially, thus slowing down the parallel execution. Although there is some potential for optimizing the FARM implementation, we stopped the development and focused on the distributed-object version of FLEX. The parallelization of FLEX using the distributed objects paradigm turned out to be a lot more promising. Figure 10 shows the absolute execution times of the considered benchmarks. The names of the benchmarks are composed as in Figure 9. Note that the execution times in Figure 10 are measured with an accuracy of • seconds. The sequential execution times (entries in the '1' row) for several benchmarks are not available due to the memory limitations. This means that it is not possible to calculate the relative speed-up in such a line (future tests on Transputer machines with 9[28] analyzes quantitatively the influence of the avalanche exponent on the applicability of parallel execution.

200

c10_3_2 c10_3_3 c10_3_4 c20_3_2 c20_3_3 c20_3_4 c20_5_3 c20_5_4 c40_3_2 c40_3_3 c40_3A c80_3_2 c80_3_3 c80_3_4

1.0 1.0 1.0 1.0

1 59 43 327 179

1.9 1.5 2.0 1.8

30 28 159 97 145 240 527 344 314

1.8 1.5 2.3 3.0

3 32 28 141 59 129 173 355 453 176 258 569 1032

3.3 2.4 3.1 3.1

5 18 18 105 58 58 164 173 411 137 231 467 665

3.3 2.7 3.8 4.0

8 18 16 87 45 56 70 155 185 105 141 264 225 443

947

2.8 2.5 4.4 4.8

11 21 17 74 37 66 74 141 126 111 111 319 200 336 662

3.3 2.9 4.4 4.5

15 18 15 73 40 51 68 160 189 72 87 230 181 266 583

Figure 10. Benchmark Execution Times.

more memo .ry will fill these gaps). This is the reason why we omited the speed-up figures in all but the first 4 benchmarks. The table shows high speed-ups (efficiencies > 80~163for all benchmarks, if the number of objects exceeds the number of processors by a certain factor (between 5 and 10). This result can be interpreted by the perspective of Section 3, where we saw that network efficiently is dependent on the number of pending propagations in the network. If this number is too low, few processing nodes are busy, resulting in a bad network efficiency. Within [28] the quantitative analysis shows that the propagation-processor ratio is more relevant to system performance than the overhead caused by message passing. 1~ It also indicates how these problems can be overcome, allowing for even larger networks. A major problem for all distributed systems is the balance of load inside the network. Within distributed FLEX each object represents a potential load. Unfortunately, the presence of objects is only a statistical measure for load, while the actual distribution depends on runtime conditions. The illustration in Figure 11 depicts execution of a benchmark with an uneven load distribution. The Transputers 2 and 4 slow down the overall performance. It is easy to see that the network is quite busy during the first half of the execution time (ca. 75~163 efficiency). At the second half, all object servers have terminated, except two (ca. 25% efficiency). This leads to a reduction of the overall efficiency to ca. 50~163 and explains the variation of the results in Figure 10. The necessa~, optimization of the uneven distribution of processor load o