This content was uploaded by our users and we assume good faith they have the permission to share this book. If you own the copyright to this book and it is wrongfully on our website, we offer a simple DMCA procedure to remove your content from our site. Start by pressing the button below!
Lecture Notes in Computer Science Edited by G. Goos, J. Hartmanis and J. van Leeuwen
1413
Barbara Pernici Costantino Thanos (Eds.)
Advanced Information Systems Engineering 10th International Conference, CAiSE' 98 Pisa, Italy, June 8-12, 1998 Proceedings
~ Springer
Series Editors Gerhard Goos, Karlsruhe University, Germany Juris Hartmanis, Cornell University, NY, USA Jan van Leeuwen, Utrecht University, The Netherlands Volume Editors Barbara Pernici Politecnico di Milano Piazza Leonardo da Vinci 32, 1-20133 Milan, Italy E-mail: pernici @elet.polimi.it Costantino Thanos Istituto di Elaborazione della Informazione - CNR Via S. Maria 46, 1-56126 Pisa, Italy E-mail: thanos @iei.pi.cnr.it Cataloging-in-Publication data applied for Die Deutsche Bibliothek - CIP-Einheitsaufnahme
Advanced information systems engineering : 10th international conference ; proceedings / CAiSE '98, Pisa, Italy, June 8 - 12, 1998. Barbara Pernici ; Constantino Thanos (ed.). - Berlin ; Heidelberg ; New York ; Barcelona ; Budapest ; Hong Kong ; London ; Milan ; Paris ; Santa Clara ; Singapore ; Tokyo : Springer, 1998 (Lecture notes in computer science ; Vol. 1413) ISBN 3-540-64556-X
CR Subject Classification (1991): H.2, D.2, H.5.2-3, H.1, K.6.3-4, H.3.3, 1.2 ISSN 0302-9743 ISBN 3-540-64556-X Springer-Verlag Berlin Heidelberg New York This work is subject to copyright. All rights are reserved, whether the whole or part of the material is concerned, specifically the rights of translation, reprinting, re-use of illustrations, recitation, broadcasting, reproduction on microfilms or in any other way, and storage in data banks. Duplication of this publication or parts thereof is permitted only under the provisions of the German Copyright Law of September 9, 1965, in its current version, and permission for use must always be obtained from Springer-Verlag. Violations are liable for prosecuUon under the German Copyright Law. 9 Springer-Verlag Berlin Heidelberg 1998 Printed in Germany Typesetting: Camera-ready by author SPIN 10637215 06/3142 - 5 4 3 2 1 0
Printed on acid-free paper
Preface CAiSE*98 was the 10th m the series of CAiSE conferences and was promoted by the CAISE Advisory Committee. The aim of this series of cont?rences is to give researchers and professionals from universities, research centres, industry and government the opportumty to meet annually to discuss evolving research issues and applications in the field of information systems engineenng; to assist young researchers in establishing relationships with senior scientists in their areas of interest: to enable review and discussion of research under way in the world of information systems engmeenng: to stimulate researchers, especially young scientists, to explore new areas of interest in information systems development. A special theme of the CAiSE*98 Conference was "bzformation Systems Engh~eering hz Public Admbffstrations". Information systems in the public adrrnnlstration domain present particular features in terms of users, goals, and requirements that charactense them with respect to other business reformation systems. In addition, an important issue concerns harmonisation of public administration procedures and data, and public administrations in a multinational community, for instance, for exchanging data of common interest. An international programme committee was set up for this conference with representatives from 19 countries. It received 102 full papers, each paper was evaluated by 3 reviewers, and 21 papers of high academic quality were selected for presentation. These papers cover a wide spectrum of topics which include design, data warehouses, workflow management and groupware, reuse, and web-based application design. Three invited talks and poster and panel sessions were also included in the programme. The conference was preceded and accompanied by a number of scientific events. Six tutorials on hot topics (Multimedia Databases, Distributed Heterogeneous Information Services, Introduction to the Unified Modelling Language. Electronic Commerce: State of the Art and Research Directions, Web-Based Information Systems: Design and Maintenance, Component Databases) were given by well known experts in the field. Two one-day tutorials preceded the conference while the four half-day ones were included in the conference programme. Five thematic two-day workshops (Innovative Internet Information Systems, Evaluation of Modelling Methods in Systems Analysis and Design, Requirements Engineenng: Foundation for Software Quality, Component-based Information Systems Engineering, Doctoral Consortium) preceded the conference. We would like here to thank all those institutions and individuals who made this conference possible: CNR, Politecnico di Milano, CAiSE Advisory Committee, the programme committee members, the invited speakers, the tutorialists, the panellists, the poster presenters, the sponsors, and of course all the participants. March 1998
Barbara Pernici Costantino Thanos
vii
CAiSE*98 Conference Organisation General Conference Chair Costantlno Thanos IEI-CNR
Programme Chair Barbara Pernici Politecnico di Milano
Organising Chair Gianni Mainetto CNUCE-CNR
Programme C o m m i t t e e Peter Aiken, USA Carlo Batini, Italy Sjaak Brinkkemper, The Netherlands Janis Bubenko, Sweden Silvana Castano, Italy Panos Constantopoulos, Greece Nina Edelweiss, Brazil Marcel Franckson, France Mariagrazia Fugini, Italy Andreas Geppert, Switzerland Paul Grefen, The Netherlands Georges Grosz, France Juhani Iivari, Finland Yannis Ioannidis, USA Matthias Jarke, Germany Keith Jeffery, United Kingdom Christian S. Jensen, Denmark Hannu Kangassalo, Finland Gerti Kappel, Austria Kamal Karlapalem, Hong Kong Frederick Lochovsky, Hong Kong Pericles Loucopoulos, United Kingdom Katie Lyytinen, Finland
Neil A.M. Maiden, United Kingdom Robert Meersman, Belgmm Carlo Meghini, Italy Alberto Mendelzon, Canada John Mylopoulos, Canada Antom Olive, Spain Micael Papazoglou, The Netherlands Klaus Pohl, Germany Naveen Prakash, India Colette Rolland, France Matti Rossi, Finland Gabriel Sanchez Gutierrez, Spain Michel Scholl, France Arie Segev, USA Timos Sellis, Greece Amilcar Sernadas, Portugal Keng Siau, USA Richard Snodgrass, USA Ame Solvberg, Norway Stefano Spaccapietra, Switzerland Babis Theodoulidis, United Kingdom Yannis Vassiliou, Greece Yair Wand, Canada Roel Wieringa, The Netherlands
VIII
Additional Referees
Anastassia Ailamak~ Bernd Amann Anastasia Analyti Camille Ben Achour Antonia Bertolino Michael Boehten Enk Boertjes Terje Brasethvik Steinar Carlsen Fabio Casati Donatella Castelli Vassilis Christophldes Gerhard Chroust Ernesto Damiani Valeria De Antonellis Sabrina De Capitani di Vimercati Dirk Deridder Ruxandra Domenig Christian Falkowski Babak Amin Farshchlan Piero Fraternali Chiara Francalanci Christophe Gnaho Wim Goedefroy Paula Gouveia Sari Hakkarainen Tom Henriksen Patrick C.K. Hung Jouni Huotari Le Van Huu Willem Jonker Elisabeth Kapsammer
Panos Kardasls Vagelio Kavakh Minna Koskmen Markus Kradolfer Spyros Ligoudistlanos Mario Loffredo Penm Marttlin Thomas Meyer Isabelle Mirbel Vincent Motte Selmin Nurcan Fabio Patemo' Torben Bach Pedersen Nikos Prekas Jaime Ramos Robert Rump Werner Retschltzegger Pasquale Savino Pierre-Yves Schobbens Michael Schrefl Yannis Stavrakas Samira Si-Said Carine Souveyet David Spelt Peter Stuer Kristlan Torp Nectaria Tryfona Alejandro A. Vaisman Panos Vassiliadis Jochem Vonk Rob L.W. van de Weg Jef Wijsen
Invited Speech A comprehensive view of process engineering C. Rolland
Design I Aligning legacy information systems to business processes P. Kardasis, P. Loucopoulos Automated reverse engineering of legacy 4GL information system applications using the ITOC workbench Z V. Harrison, W. M. Lim Adapting function points to object oriented information systems G. Antoniol, F. Calzolari, L. Cristoforetti, R. Fiutem, G. Caldiera
25
41
59
Data Warehouses and Extensible DBMS Global cache management for multi-class workloads in data warehouses S. Jin, X. Sun
77
Architecture and quality in data warehouses M. Jarke, M.A. Jeusfeld, C. Quix, P. Vassiliadis
93
OMS/Java: Model extensibility of OODBMS for advanced application domains A. Steiner, A. Kobler, M.C. Norrie
115
Workflow Management and Groupware An environment for designing exceptions in workflows F. Casati, M.G. Fugini, L Mirbel
139
Automating handover in dynamic workflow environments C. Liu, M.E. Orlowska, H. Li
159
Document-centric groupware for distributed governmental agencies D.A. Tietze, A. Bapat, R. Reinema
173
Reuse Specifying the reuse context of scenario method chunks C. Rolland, V. Plihon, J. Ralyt~ Change analysis and management in a reuse-onented software development setting W. Lain A filter-mechanism for method-driven trace capture R. DOmges, K. Pohl, K. Schreck
191
219
237
Application Design and Web Subject-based organization of the information space in multi-database networks M.P. Papazoglou, S. Milliner
251
MUSE - An interactive networked multimedia applications specification environment with E-LOTOS translator L. P. Gaspary, M.J.B. Almeida
273
Information extraction and database techniques: A user-oriented approach to querying the Web Z Lacroix, A. Sahuguet, R. Chandrasekar
289
Industrial Experiences Goal-driven business process analysis - Application in electricity deregulation V. Kavakli, P. Loucopoulos
305
Real-time information system for risk management on motorways T. Tanzi, S. Servigne, R. Guiol
325
Describing business processes with a guided use case approach S. Nurcan, G. Grosz, C. Souveyet
339
Design II Building quality into case-based reasoning systems I. Jurisica, B.A. Nixon
363
Assembly techniques for method engineering S. Brinkkemper, M. Saeki, F. HarTnsen
381
Formalizing materialization using a metaclass approach M. Dahchour
401
Author Index
423
A Comprehensive View of Process Engineering Colette Rolland University Paris-1 Sorbonne, 17, rue de la Sorbonne, 75005 Paris cedex 5, FRANCE email : [email protected]
Abstract. The paper proposes a faceted framework to understand and classify issues in system development process engineering. The framework identifies four different but complementary view-points. Each view allows us to capture a particular aspect of process engineering. Inter-relationships between these aspects allow us to show the influence that one aspect has on another. In order to study, understand and classify a particular aspect of process engineering in its diversity we associate a set of facets with each aspect. The paper uses the framework to raise questions, problems and research issues in the field.
1. INTRODUCTION Process engineering is considered today as a key issue by both the software engineering and information systems engineering communities. Recent interest in process engineering is part of the shift of focus from the product to the process view of systems development. Process engineering is a rather new research area. Consequently there is no consensus on, for example, what would be a good formalism to represent processes in, or, even, on what the final objectives of process engineering are [2] . However, there is already considerable evidence for believing that there shall be both, improved productivity of the software systems industry and improved systems quality, as a result of improved development processes [14], [2] and [31]. Studies of software development practices [37], however, demonstrate that we know very little about the development process. Thus, to realise the promise of systems development processes, there is a great need [14] for a conceptual process engineering framework. In this paper we consider process engineering from four different, but complementary, view-points. Each view allows us to capture a particular aspect of process engineering. Inter-relationships between these aspects allow us to show the influence that one aspect has on another. In order to study, understand and classify a particular aspect of process engineering in its diversity we associate a set of facets with each aspect. For example, in the development view, where the concern is with the way in which process models are developed, it is possible to turn to (a) the facet called construction approach to understand how a process model can be constructed, (b) the construction technique facet to understand how it can be engineered, (c) the change support facet to see how flexible the process model is etc.. B. Pernici and C. Thanos (Eds.): CAiSE’98, LNCS 1413, pp. 1-24, 1998. Copyright Springer-Verlag Berlin Heidelberg 1998
2
Colette Rolland
Facets have been proposed by [50] for classifying reusable components. They have also been used by [59] in requirements engineering for understanding and classifying scenario based approaches. When used in process engineering, a facet provides a means for classification. For example, the coverage facet of the system world (see section 5 below) helps in classifying process models according to the underlying paradigm used: activity-oriented, product-oriented, decision-oriented or contextual. Each facet is measured by a set of relevant attributes. For instance, the description facet is measured by two attributes, the form and the notation attributes. Attributes have values which are defined in a domain. A domain may be a predefined type (INTEGER, BOOLEAN ...), an enumerated type (ENUM {x, y, z}), or a structured type (SET or TUPLE). We use the four worlds framework as a baseline and attach (a) a view of process engineering to each of its worlds and (b) a set of facets to each view. As a result, it is possible to identify and investigate four major view points of process engineering: what are processes, how are they represented, how can their representation be developed and used, and, finally, what does process engineering achieve. The multi-facet, multi-view approach adopted here makes it possible to look at process engineering in a comprehensive manner: - facets provide an in-depth description of each aspect of process engineering whereas aspects give a view of process engineering in all its diversity. - relationships between facets help in understanding the implications of one view on another.
2. THE FOUR-WORLDS FRAMEWORK The four worlds framework originally proposed for system engineering has proved its efficiency in enhancing the understanding of various engineering disciplines, information systems engineering [29], requirements engineering [30], and method engineering [58]. It can also help in understanding the field of process engineering which consists of applying engineering approaches, techniques, and tools to the construction of process models. How are processes of the subject world used
USAGE WORLD
SUBJECT WORLD
User interfaces
Justification of development goals
DEVELOPMENT WORLD
How does the process model represent the subject world
SYSTEM WORLD
design decisions
Fig. 1. The four worlds of process engineering
A Comprehensive View of Process Engineering
3
In the original system engineering framework (Fig. 1.), the subject world contains knowledge of the domain about which the proposed IS has to provide information. It contains real-world objects which become the subject matter for system modelling. The system world includes specifications at different levels of detail of what the system does. It holds the modelled entities, events, processes, etc. of the subject world as well as the mapping onto design specifications and implementations. The usage world describes the organisational environment of the information system, i.e. the activity of agents and how the system is used to achieve work, including the stakeholders who are system owners and users. The usage world deals with the intentional aspects of the IS to be built whereas the subject world refers to the domain it shall represent. The development world focuses on the entities and activities which arise as part of the engineering process itself. It contains the processes which create the information system i.e. processes which involve analysis and understanding of knowledge contained in the other worlds and representation of that knowledge. For our purposes, we identify the subject world as the world of processes. The system world deals with the representation of processes through process models. In the usage world we will investigate the reasons, the rationale for process engineering and relate the objectives of the users to the process models that can best meet these objectives. The development world deals with the process of constructing process models. This process is a meta-process in the sense that it supports the construction of processes, which in turn, will support the development of systems. The way the process might be supported by a tool environment is also a concern of this world. The paper uses the four worlds to present the state of art in process engineering and to raise questions, problems and research issues in the field. It comprises four sections, each of these relating to one of the world. This allows us to discuss in a focused manner the different concerns of process engineering : the definitions of processes, their representations, the way of developing these representations, and the rationale for using these representations. This is done in the subject, system, development, and usage worlds respectively.
3. THE SUBJECT WORLD Our Universe of Discourse is the world of processes. In this world, it is of interest to look at the notion of a process and its nature. A process is performed to produce a product. It has been described in the information systems area [41] as the route to be followed to reach a product. This basic notion has been extended by [44] who looks upon a product as a point in three-dimensional space comprising of the agreement, specification, and representation dimensions. Starting from some initial position in this space, the product moves through a succession of locations before a final position is reached. This final position corresponds to the desired product. The process then can be considered to be the
4
Colette Rolland
route starting from the initial product position and going through the succession of intermediate positions till the final product position is reached. The term process has been defined differently in different coverage (see section V below for the notion of coverage). In the activity-oriented coverage it is defined as a related set of activities conducted for the specific purpose of product definition. In [68] it is defined as "a set of partially ordered steps intended to reach a goal" and a process step is itself an atomic action of a process that has no externally visible substructure. In the product-oriented coverage, a process is a series of activities that cause successive product transformations to reach the desired product. [21], [26] and [36] are three examples of definitions conforming to this view. In the decisionoriented coverage, a process is defined as a set of related decisions conducted for the specific purpose of product definition. This view has been developed, for instance in IBIS [46], DAIDA[29] and [60]. Finally, in the coverage called context, a process is a sequence of contexts causing successive product transformations under the influence of a decision taken in a context [30]. More intrinsically processes can be of different kinds. These various definitions reflect the multiple view points of the community about what is a process. However, these view points correspond to the various ways in which a process can be modelled and we will deal with in the system world. Strategic processes are those that investigate alternative ways of doing a thing and eventually, produce a plan for doing it. There are many ways of doing the same thing and the best way, the one most suited to the situation at hand has to be found. Such processes are creative and alternative generation and selection from an alternative are very critical activities here. Tactical processes are those which help in the achievement of a plan. As their name implies they are more concerned with the tactics to be adopted for actual plan achievement than with the development of a plan of achievement. Implementation processes are the lowest level processes. They are directly concerned with the details of the what and how of plan implementation. Thus, the subject world can be characterised by a facet having only one attribute called Nature defined as Nature: ENUM{strategic, tactical, implementation} As one can expect, we shall see below how the nature of the processes handled will influence the choice of a model adequate for their representation.
4. THE USAGE WORLD The usage world is where the goals of process use are established and, consequently, the range of facilities required for process performance are determined. The usage
A Comprehensive View of Process Engineering
5
world can be viewed [14] as composed of three interacting domains : a process model domain, a process performance domain, and a process model enactment domain (Fig. 2.). Process Performance Domain
Process Model Domain
Human agents, Activities
Model Fragments
Process Improvement, Capitalisation of Experience
Enactement Creation
Guidance Monitoring/Controling
Feedback
Process Model Enactement Domain Enactement Mechanism
Fig. 2. Process domains
The process model domain contains process models. A process model describes the common properties of a class of processes having the same nature. The process performance domain deals with the actual activities conducted by human agents and machines, in the course of a project. Some will be executed by software tools; others will consist of human thinking, writing, exchanging ideas, and taking decisions through formal and informal interactions between members of the project team. All these activities must be supported by the process model. The process model enactment domain is concerned with the features needed to support process performance governed by the process model. These features support, guide, or enforce performance of the process in a way consistent with the process model. The three domains interact with each other in different ways. Firstly, the process model influences the way in which the process is performed. Actual performance then corresponds to some extent to the model of how it should be performed. Secondly, the course of enactment may need to be contingent on events arising from actual process performance. Therefore, the actual process will be different from the theoretical instantiation of the process model. This leads to the idea of feedback from process trace to process model, thereby allowing its improvement. This leads to a view of the usage world as imposing strong requirements on the way processes will be performed, the nature of process models used and the way in which these process models will be developed and changed. The purpose assigned to the process model has to be determined by the usage world. This is captured below in the facet, Purpose. Since the way processes are performed changes with time, it is the
6
Colette Rolland
duty of the organisation to define their process management policy. This is captured in the facet, Process Management Policy. 4.1 PURPOSES A synthesis of proposals from the software engineering field [36], [11], the information system community [7], [48], [56], and the system design community [51], [46], show three main aims of process models: - descriptive, to record trace what actually happens during a process, - prescriptive, to define desired processes and how they should/could/might be performed, - explanatory, to provide explanations about the rationale of processes. A descriptive purpose takes the point of view of an external observer who looks at the way a process has been performed and determines the improvements that have to be made to make it perform more effectively or efficiently. The prescriptive purpose lays down rules, guidelines, and behaviour patterns which, if followed, would lead to the desired process performance. The prescriptive purpose lies in a range from strict enforcement to flexible guidance. In the former the performance of the process must follow the prescription whereas in the latter the prescription is such that it can accommodate a large number of ways in which the process can proceed. Guidance shifts the emphasis away from task performance to goal achievement. Therefore, there can be two types of guidance, point and flow [63]. Point guidance provides help in the achievement of a given goal whereas flow guidance helps in identifying the next goal in order for the process to proceed. The explanatory purpose is important in those processes where several possible courses of action are open and each of these has to be explored and evaluated based on rational arguments. Such traces establish an explicit link between processes and the requirements that they are to fulfil. The descriptive and explanatory purposes have been accorded a lot of attention in the recent past. This is because of the need to keep track of process knowledge and to support change [20], [51]. To take this to the extreme, it is difficult to visualise any process, strategic, tactical, or implementation (see Subject World), without a descriptive and/or explanatory purpose behind them. Specifically, if prescription is to be provided to strategic processes, then flexible guidance is clearly more appropriate than process enforcement. This is because strategic processes are often creative and require human co-operation. This makes most software process models inappropriate for strategic processes because [19] their basic property is enforcement of constraints (prescriptions and even proscriptions). However, in tactical or implementation processes of the Subject World that follow plans relatively more strictly and which are less creative and mercurial, varying
A Comprehensive View of Process Engineering
7
shades of process enforcement ranging from mechanical enforcement with limited guidance to complete automation may be found useful. A process engineering approach can be classified according to the role it aims to play in the facet called Purpose which has the three following attributes : Prescriptive: ENUM {enforcement, guidance} Descriptive: BOOLEAN Explanatory: BOOLEAN 4.2 PROCESS MANAGEMENT POLICY Processes change with time and so do the process models underlying them. Thus, new processes and models may have to be built and existing ones improved. There is need to have a well-defined organisational policy to handle this change. This policy can either accept change continuously as it occurs or accept it as one-shot, radical change. Radical change applies in situations where organisations need to define a process management policy from scratch. The former applies when need is felt to harmonise heterogeneous process practices or when a bottom-up approach is systematically applied to move up in the levels of maturity in the CMM [26] framework. . Strategic processes are highly unstable. The process proceeds by analogy with other similar processes and reuses experience and knowledge of their stakeholders. This reuse is continuous and operates so long as the process lasts. It is today implicitly done by individual human agents performing the process but, perhaps, in future, it shall be necessary to have reuse as a process management policy of the organisation. However, it remains to be conclusively shown that process practice reuse is cost effective in an organisational setting. The foregoing is captured by the two attributes change and reuse of the Process Management Policy facet Change: ENUM{continuous, radical} Reuse: BOOLEAN
5. THE SYSTEM WORLD If the subject world is the world of processes then the system world is the one of their representations. The interest in this world is in a) what is to be represented b) at what level of abstraction c) how is it represented d) what properties should the representation have. The facet contents, of the system world deals with (a), the abstraction facet deals with (b), the description facet deals with (c), and finally, the modularization facet captures the properties of the representation. We develop each of these below.
8
Colette Rolland
5.1 ABSTRACTION Processes of the same nature are classified together into a process model. Thus, a process model is a description of a process at the type level. Since the process model is at the type level, a process is an instantiation1 of it. The same process model is used repeatedly for the development of many applications and thus, has many instantiations. As stated in section 4, one possible use of a process model is to prescribe "how things must/should/could be done" in contrast to the process itself which is really what happens. A process model is more or less a rough anticipation of what the process will look like. What the process shall, indeed, be will, however, be determined during actual system development. A process meta-model is a description at the type level of a process model. A process model is, thus, an instantiation of a process meta-model. Obviously, a meta-model can be instantiated several times in order to define various process models. A process meta-model is at the meta-type level with respect to a process. It plays a role similar to a theory in the Theory of Plans [66] from which plans can be generated (the process models) and executed (the processes). The abstraction facet captures the levels at which the model is expressed and the corresponding attribute takes on values from the enumerated domain { type, metatype}. The well known models like the waterfall [61] and spiral models [6] are the type level whereas the Nature process theory [55] is at the meta-type level.
5.2 CONTENTS The concern of this facet is with the contents of the process model/meta-model. These contents are determined by the system of concepts in terms of which processes are represented and by the granularity of these representations. These two aspects are dealt with by the coverage and granularity attributes respectively. 5.2.1 Coverage According to Dowson [13], process models can be classified into three groups of models: - activity-oriented, - product-oriented, and - decision-oriented. 1
A. Finkelstein in[18] points out the various meaning of the widely used term "instantiation" in the software engineering community. We relate here to the classical idea of creating instances from a type/class definition
A Comprehensive View of Process Engineering
9
Since this classification was made, a new group called the contextual model has also emerged. Activity-oriented The activity-oriented models concentrate on the activities performed in producing a product and their ordering. These process models are sequential in nature and adopt the Cartesian, functional decomposition approach. They provide a frame for manual management of projects developed in a linear fashion. The first widely used process model, the Waterfall model [61], falls into this category. Its widespread acceptance has led to life-cycle descriptions being most often treated as linear sequences where crucial aspects of the process such as feedback loops and iteration are not represented [6], [10] and [11]. These models are well suited to model implementation processes. The strong emphasis on an activity incurs some risks of neglecting the influence of product structure on the process. Further, they are unable to support flexible prescriptive guidance but only process model enforcement. The linear view of activity decomposition seems inadequate to model creative processes because it is not possible to consider all contingencies. Activity-oriented representations cannot incorporate the rationale underlying the process and therefore do not permit reasoning about engineering choices based on existing conditions. It is unrealistic to plan what will happen in such a process in an entirely sequential manner. Finally, the linear view is inadequate for process models which have to support backtracking, reuse of previous designs and parallel engineering. Product-oriented Product-oriented process models, in a manner similar to activity-oriented ones, are centred around the notion of activity but, additionally, link activities to their output : the product. The ViewPoints model [18] and the process model proposed in the European Software Factory (ESF) project [21] belong to this category. Product-oriented models couple the product state to the activities that generate this state. They visualise the process as a state transition diagram. Since product-oriented models adopt the notion of an activity, they suffer from the same difficulties as the activity-oriented models considered above. However, due to their product-activity coupling they are useful for tracing the transformations performed and their resulting products. However for strategic processes it is difficult, if not impossible, to write down a realistic state-transition diagram. Decision-oriented The successive transformations of the product caused by a process are looked upon, in decision-oriented models, as consequences of decisions. The process models of the DAIDA project [29], [46] and [51] fall into this category. These models emphasise the concept of an "Intention " at the expense of an activity.
10
Colette Rolland
Decision-oriented models can be used for both, strategic as well as tactical processes. The strength of the decision-oriented approach is its ability to cover more aspects of a process than can be done by the two other kinds. Decision-oriented models are not only able to explain how the process proceeds but also why it proceeds. Therefore, decision-oriented process models are particularly well suited to strategic processes, for supporting explanatory tracing and prescriptive guidance. This is because of their ability to (a) guide the decision making process (b) help in reasoning about the rationale behind decisions,(c) support the deliberation underlying the decision process itself and (d) keep a trace of the happenings of a process and their rationale. Contextual Models Contextual models as found in the Nature process theory [8], and in the F3 project [54] look upon each process as being in a subjectively perceived situation upon which is looked upon with some specific intention. The work to be done next depends on both the situation and the intention i.e. it depends on the context existing at this moment. Contextual process models strongly couple the context of a decision to the decision itself. It makes the notion of a context, the coupling of a situation and the decision, central to process modelling. Decisions are applied to the situation in which the process currently is, in order to change that situation to the desired new one. In this respect, the contextual approach has some similarity with the planning paradigm that has emerged from Artificial Intelligence and with projects based on the planning paradigm such as GRAPPLE [25]. Since the contextual models adopt the notion of a decision, all the properties of decision-oriented models mentioned earlier are applicable to them. Further, due to the strong relationship between the situation and the decision, only those decisions which are appropriate in the situation at hand are of interest. This helps in focusing guidance, tracing and explanation to specific process situations. Process models can be classified within the facet Contents, by giving values to the attribute, coverage, Coverage: ENUM{activity, product, decision, context} 5.2.2 Granularity Most traditional process models are large-grained descriptions of the product lifecycle. On the other hand, there are very fine-grained models. For example specifying that after a source code file is edited, it should be recompiled [33]. Recently, hybrid formalisms that use different notations for large-grain and small-grain aspects of process such as PROCESS WEAVER [17], have been developed.
A Comprehensive View of Process Engineering
11
The nature of granularity needed is dependent on the situation at hand. Granularity affects the kind of guidance, explanation and trace that can be provided. High granularity limits these to a rather coarse level of detail whereas fine granularity provides more detailed capability. Process models should, ideally, provide a wide range of granularity. This shall allow a movement from large grains to fine grains along a continuum. Therefore, the granularity attribute takes on values from SET(ENUM{large, fine, variable}). 5.3 THE DESCRIPTION FACET The description facet is concerned with the form of the process representation and the level of formality of the language used to describe the process model. These correspond to the form and notation attributes of this facet. 5.3.1 Form The form attribute is concerned with style of the process representation. There are three identified forms, scripts, programs, and hypertext. Osterweil [42] has proposed the view that software process models should take the form of a program as different from process scripts. Process scripts are interactively used by humans as against process programs which are enacted by a machine [35]. They support non determinism whereas process programs can, at best, support process deviation under pre-defined constraints [9]. The hypertext style of process representation is a network of links between the different aspects of a process, such as product parts, decisions, arguments, issues, etc. A relationship can be established between form and the purpose facets of the Usage World. Scripts and programs are two styles which may be applicable to prescriptive purposes whereas hypertext is well suited to descriptive and explanatory purposes. Strict enforcement of the prescriptive purpose can clearly be represented in process programs whereas flexible guidance requires the process model to be represented in process scripts. Descriptive and explanatory purposes require the establishment of relationships between different elements of a process trace. These relationships are well articulated as hypertext links. The form attribute of the description facet takes on values from ENUM{script, program, hypertext} 5.3.2 Notation Process models underlying information systems practice have traditionally used informal notations such as natural languages or diagrams with informal semantics. On the other hand, in software engineering, more formal software process models (see [2], [11], [19] for an overview) have been used. This formality relates to underlying
12
Colette Rolland
programming languages : Smalltalk for E3 [19], various Prolog dialects for EPOS [28], Oikos [1], and PEACE [19], PS-Algol for PWI [19]. A formal notation is required to support the verification of the expected properties of the process model and validation of the process model using for instance, simulation or enactment techniques. The use of informal notations has made it difficult for process models to be followed systematically. Formal or semi-formal notations make these efforts considerably more effective. Formal notations are necessary for providing automatic enactment support. The notation attribute helps classifying process models by one of the three values of the following enumeration: Notation: ENUM{formal, semi-formal, informal} 5.4 MODULARIZATION Early processes were monolithic. However, there is a shift towards modular process structure in this decade. We introduce a Boolean valued attribute called Presence in the modularization facet to distinguish between monolithic and modular methods.
One proposal for modularization [23] is that of fragments. A fragment can be either a product fragment or a process fragment. The drawback of the fragment based approach is the over-emphasis on the product fragment resulting in under developed meta-process modelling. The proposal of [52], [53], is to tightly couple the product and process aspects of processes into contexts. A Context is a couple <situation, decision>, where the decision part represents the choice an IS developer can make at a moment in the engineering process and the situation is defined as the part of the product it makes sense to make a decision on. Process modules can be looked upon according to two other perspectives : abstraction and aggregation. Rolland [55] has defined aggregates called process chunks as hierarchies of contexts. A chunk prescribes the way to proceed in the situation identified by the context at the root of its context hierarchy. This allows the decision of the root context to be taken in this situation. [65] proposes two kinds of aggregated modules called route map and fragments respectively. A route map refers to strategies such as delivery strategies, developmental strategies, realisation strategies etc., activities and products concerning system development as well as project management. The fragment is a coherent part of a process for system development or project management. Fragments may be linked to a route map which may establish a complete project approach.
A Comprehensive View of Process Engineering
13
Abstraction is used to capture generic laws governing the construction of different but similar process modules. Generic process modules can take the form [56] of framework or pattern. A framework models the commonality between modules of different process models but for the same type of application. A pattern models a common behaviour in process construction. It is generic in the sense that it is used every time a process model is constructed. Both terms have been chosen by analogy with reuse approaches in the object oriented area. Patterns are there defined as solutions to generic problems which arise in many applications [22], [49] whereas a framework is application domain dependent [67], [32].
Classification along the modularization facet comes down to giving values to the two following attributes: Presence: BOOLEAN Nature: SET( ENUM{primitive, aggregate, generic}
6. THE DEVELOPMENT WORLD The development world deals with two issues - the process of constructing process models, and - enactment of processes. The process of constructing process models is a meta-process, it is the process behind the process used to construct processes for building information systems products. The development world deals with meta-processes so as to improve process models and to make them evolve. The second issue is that of process enactment. The development world is also concerned with the way in which process models can be constructed and process enactment support provided. That is, the tool environment needed to support process performance is also a concern of this world. Thus, the facets of interest in this world are construction approach, construction technique, enactment support, and change support . 6.1 CONSTRUCTION APPROACH In a manner analogous to that of Harmsen [23] one can organise construction approaches in a spectrum ranging from 'low' flexibility to 'high'. At the 'low' end of this spectrum are rigid approach whereas at the 'high' end is modular approach. Rigid approaches lead to process models that are completely pre-defined and leave little scope for adapting them to the situation at hand. On the other hand, contingency approaches allow the modification and augmentation of models to make them fit to a given situation. There are at least two ways by which contingency approaches can be realised. The first one is the production of contingency process models that is, situation-specific
14
Colette Rolland
models for certain types of organisational settings. This presents process engineering as the selection of a model within a panel of contingency process models. In the second one process engineering is used to support the selection and the assembly of process components to construct process models ‘on-the-fly’. The foregoing suggests that construction approach should be classified as : Construction approach: ENUM{contingency, on-the-fly, rigid} The construction approach adopted in the development world has a strong impact on the modularization facet and granularity attribute of the system world. Whereas the rigid approach can be associated to monolithic models, contingency and on-the-fly approaches require modular process models. The contingency approach is well suited to support capitalisation of 'good practice' into process chunks in a stable environment. Instead 'on-the fly' approaches are adapted to the dynamic recognition and use of chunks and patterns. 6.2 CONSTRUCTION TECHNIQUE Within the broad construction approach adopted for constructing process models, a number of techniques for construction are available. Construction techniques used in the information systems area have developed independently of those in software engineering. In information systems, construction techniques exploit the notion of a meta-model and the two principal techniques used are those of instantiation and assembly. In software engineering the main construction technique used today is language-based. However, early techniques in both, information systems and software engineering were based on the experience of process engineers and were, therefore, ad-hoc in nature. We comment the attributes values in turn. 6.2.1 Instantiation Given that new process models shall be constructed very often, the question is how we can increase the productivity of process engineers and improve the quality of the models they produce. One way of doing this is to identify the common, generic features of process models and represent them in a system of concepts. Such a representation has the potential to 'generate' all process models that share these features. This potential is realised when a generation technique is defined whose application results in the desired process model. Thus, there are two key issues here - the identification of the system of generic concepts - the instantiation technique. The first of these is resolved by the definition of a process meta-model whereas the second issue is resolved by deriving process models from this process meta-model through instantiation. A number of advantages flows from this: 1) The exploitation of the meta-model helps us to define a wide range of process models. 2) It makes the activity of defining process models systematic and versatile. 3) It forces us to look for and introduce, in the process meta-model, generic solutions to problems and this makes the derived process models inherit the solution
A Comprehensive View of Process Engineering
15
characteristics. Under the instantiation approach, the crucial issue in process modelling is no longer the process model but the process meta-model. This means that the onus of providing a process model with the required characteristics shifts from the process model developer to the process meta-model developer. The instantiation technique has been used, for example, in NATURE [52], [53], [53], [56]. The process engineer must define the instances of contexts and relationships that comprise the process model of interest. It has been utilised to build the repository of Computer Aided method Engineering environments [34], [24], [38], [62]. 6.2.2 Language The software engineering community has used different languages for expressing process models like Smalltalk for E3 [19], various Prolog dialects for EPOS [28], Oikos [1], and PEACE [19], PS-Algol for PWI [19]. Different computational paradigms have also been used, for example, Petri nets in EPOS [28] and SPADE [3], rule based paradigm in MERLIN [16], ALF [5], Marvel [33], EPOS [28], and triggers in ADELE [4] and MVP-L [19]. There is a relationship between the construction technique and the form facet in the system world. Indeed, languages are typically related to process programs whereas instantiation techniques have been used to construct process scripts. 6.2.3 Assembly The assembly technique relies on the availability of process components in a process repository. Assuming that process components exist in a process repository, the question now is "how to deliver the relevant process components to the user?" The process community has been looking at this question in two ways : first, by promoting a global analysis of the project on hand based on contingency criteria and, secondly, by associating descriptors to components in order to ease the retrieval of components meeting the requirements of the user. Therefore in the former the project situation is at a very global level whereas in the latter the descriptors of process components support local matching with the situation at hand. [65] is an example of the first approach. This approach has been tried out in nine nonstandard projects of the systems development department of a bank organisation. The second approach [57] uses the notion of descriptor [12] as a means to describe process chunks. It has been tried out to construct information systems methods [45] in NATURE and repository of scenario based approaches accessible on Internet in the CREWS project [59]. For the assembly technique to be successful, it is necessary that process models are modular. If the assembly technique is combined with the instantiation technique then the meta-model must itself be modular.
16
Colette Rolland
6.2.4 Ad-Hoc Traditional process models are expressions of the experiences of their developers. Since this experience is not formalised and is, consequently, not available as a fund of knowledge, it can be said that these process models are the result of an ad-hoc construction technique. This has two major consequences : it is not possible to know how these process models were generated, and they become dependent on the domain of experience. If process models are to be domain independent and if they are to be rapidly generable and modifiable, then we need to go away from experience based process model construction. Clearly, generation and modifiability relate to the process management policy adopted (see Usage World). Instantiation and assembly, by promoting modularization, facilitate the capitalisation of good practice and the improvement of given process models. The construction technique facet is defined as follows: Construction technique: SET(ENUM{instantiation, language, assembly, adhoc}) 6.3 ENACTMENT SUPPORT Enactment mechanisms have been mainly implemented by the software engineering community as the core of Process Centred Software environments. An enactment mechanism determines and controls the interactions between the agents performing the process so as to trace, guide, and enforce performance of the process in a way consistent with the process model. Considerable effort has been put in to provide automated execution support, automated monitoring and enforcement of software processes in process centred software environments. The reader will find in [19] a detailed presentation of ten projects in the field as well as the results of a common assessment exercise performed by the leaders of these projects. Most process centred software environments [28], [4], [3], [33] are in fact used to describe the activity of tools and to allow automatic invocation of tools [64]. Existing environments guide software engineers in the selection of the right suite of tools but they do not guide the engineering activities themselves. On the contrary, some attempts have been made in the information systems community for implementing enactment mechanisms that focus on guiding engineering activities [62] Whereas the foregoing deals with supporting the performance of application processes, there is also need to support the process of constructing process models, the meta-process. Just as other processes are represented by process models, the metaprocess shall have its own model, the meta-process model. Again, the meta-process itself needs to be performed in accordance with the meta-process model and this means that enactment support has to be provided for the performance of the metaprocess.
A Comprehensive View of Process Engineering
17
It is possible to build two separate enactment environments for dealing with process and meta-process enactment respectively. However, if the meta-process is treated as just another process then it is possible to use the same enactment mechanism to support both, the process and the meta-process. In fact this has been demonstrated in the Mentor environment[62]. Enactment mechanisms must support processes that take the form (see System World) of scripts, programs, or hypertext. When a process model is a script then the enactment mechanism provides high flexibility so as to enable human agent intervention during process performance. This intervention would be supported by guidance mechanisms which may either, proactively provide suggestions on possible decisions that could be taken or may support requests for help. In terms of the Usage World, for models which are process programs, the enactment mechanism behaves like the run-time support of programming languages. Process program enactment is program execution whereas process script enactment is model interpretation. Finally, when models are of the hypertext form then the enactment mechanism offers facilities to create links between process decisions, their rationale, and the resulting product. Since the meta-process is a process, it is possible to extend the foregoing remarks to it as well. However, as it is unlikely to completely plan out the meta-process, it would be the case that meta-process models are not treated as process programs but as process scripts only. This facet has two Boolean values attributes Process support: BOOLEAN Meta-process support: BOOLEAN 6.4 CHANGE SUPPORT The traditional practice is that if a process model does not meet requirements of the users then a new one is built. This practice causes loss of experimental knowledge which could have been used to change process models. The development world must therefore provide support for process model change. There are two different ways in which this can happen (a) process model change takes place even as the process proceeds: the process model can be adapted to specific requirements as these emerge, (b) the process model may need to be revised and improved at the end of the project: this is to benefit from the experience gained in process model use. The former is referred to as dynamic process change [15] and the latter as process improvement [36]. Different positions are taken in the software process engineering community concerning the need for dynamic changes. On the one hand, people claim that this is an essential requirement and some software process centred environments EPOS [28], E3 [19], SPADE [3], ADELE [4] try to provide solutions for it [19]. On the other
18
Colette Rolland
hand, it can be argued that a prescriptive approach to process modelling is at odds with dynamic process evolution [27]. The notion of fitness of the process has been defined in [27] as the degree to which the agents performing the process can faithfully follow the process steps it specifies. When this fitness is low then process change occurs. Thus, process model change is an indication of lack of flexibility of the model. Recent process models include the concept of process deviation and therefore control the deviation of process performance from that prescribed in the process model. There are only initial experiments in improving the process model by experiencebased learning, as suggested in the literature [26], [40], [43]. They suggest two ways of process improvement, by inductive or deductive learning. Inductive learning is based on the analysis of process deviations that are recorded in process traces. Induction improvement can be performed by a human agent who, on his own, decides the change that is needed. The agent can be supported by generalisation rules [39] that can be part of a tool based inductive learning environment [47]. In order to do inductive improvement, there must exist a mapping between elements of the trace and the concepts of the process model. Deductive learning exploits Case-based reasoning. Thus, it solves new problems by adapting solutions that have been utilised to resolve past problems [51]. Case based reasoning when applied to process performance calls for the existence of a repository of cases. Deductive process improvement aims at adding new cases in the repository by examining process performances. Deductive learning corresponds to the retaining phase of the Case based reasoning cycle which traditionally consists of the four phases (a) retrieve, (b) reuse, (c) revise, and (d) retain. Dynamic process change and process improvement are the two techniques that the Development World can offer to support the process management policies set in the Usage world. Deductive process improvement is appropriate when an organisation wants to promote the reuse of good practice in performing processes. Clearly, a process model supporting aggregates (see Modularization facet in System World). shall be well suited to provide these as cases to be stored in the repository. Inductive improvement is well suited to situations where process models are used repeatedly and can continuously be improved by learning from the deviation of actual performances. A modular structure of process models helps in relating the observed deviations to specific, localised parts of the process model components and therefore facilitate inductive improvement. The change support attribute takes one or several values among the following enumerated domain : Change support: SET(ENUM{dynamic process change, process improvement})
A Comprehensive View of Process Engineering
19
7. CONCLUDING REMARKS The subject and the usage worlds constitute the environment within which the technical view of process engineering contained in the system and development worlds lies. This embedding is the source of the inter-relationships between the facets of the four views discussed in this paper. The nature of processes and the purpose imposed by the usage world on the process engineering solution determine, to a large extent, the type of contents and description of the process models/meta-models. The choice of a particular type of content and description based on the nature of the processes guarantees that the semantics of these processes are well captured in process models/meta-models. On the other hand, selection of a content and description to meet the purpose expressed by the usage world guarantees that the process model/meta-model shall fulfil the requirements of the process stakeholders. In fact, we suggest that selection based on purpose should have primacy over that based on the nature of the process. This conclusion can be drawn by analogy with ISE approaches where it has been recognised that user needs are better met by understanding usage goals and not merely by using good semantic conceptual model. In fact, the usage world affects the system world through both, the purpose as well as the process management policy. Of the three purposes, the explanatory and the descriptive have been incorporated in process models that provide design rationale and process traceability respectively. However, the issue of providing prescriptive guidance is still open. The process management policy affects the abstraction facet as it introduces the need for abstracting process models into process meta-models. Before building meta-models which reflect old models but on a new level of abstraction, one should question the old ones. The goal of meta-modelling is not only to operationalise current process models but also to correct the general oversights and limitations of these. In a similar manner, we believe that the technical solution in the development world has to be chosen according to the purpose and the process management policy decided in the usage world. The influence of the former is clearly on the choice of the enactment mechanisms. The implication of the latter is more diverse. The policy recognises the need for managing processes in-the-large and their evolution in time. The former sets the requirements of an automated enactment support and its extension to the meta-process. The latter determines the choice of the change support and of the construction approach. There are two implications of this, organisational and technical. The absence today of organisation-wide process policies raises the need for organisations to understand the crucial role played by these policies and to define them. Such policies would, for example, encourage capitalisation of good practice, learning from experience, and development of a reuse culture. The capitalisation policy raises the technical question of how good practices can be recognised, organised and reused. Such knowledge should be available in the form of
20
Colette Rolland
chunks for later assembly. This implies the modularization of process models/metamodels. The modular approach represents a shift in the way of thinking about process representations and is likely to emerge as a major research issue in the future. The reuse culture raises the question of the genericity of process representations. Perhaps, what is needed is a corpus of both, generic process and generic meta-process knowledge in a modular form. A suggested research is therefore, the development of a domain analysis approach to identify common properties of (a) different process models and (b) different meta-process models. The evolution of process models calls for the establishment of technical enactment support for the meta-process. So far, work has concentrated on developing and experimenting with process enactment mechanisms. The research issue here is of making these mechanisms generic enough to handle both process and meta-process enactment. This paper has argued that process engineering should be usage driven. The acceptance of process engineering in organisations is, however, not entirely determined by the functionality that is needed but also by other non-functional factors such as usability, availability, security etc. This aspect of process engineering has to be addressed by the research community more vigorously.
REFERENCES 1. V; Ambriola, M. L. Jaccheri, Definition and Enactment of Oikos software entities, Proc. of the First European Workshop on Software Process Modeling, Milan, Italy, 1991 2. P. Armenise, S. Bandinelli, C. Ghezzi, A. Morzenti, A survey and assessment of software process representation formalisms Int. Journal of Software Engineering and Knowledge Engineering, Vol. 3, No. 3, 1993. 3. S. Bandinelli, A. Fugetta, S. Grigoli, Process Modelling in the large with SLANG, Proc. of the 2nd Int. Conf. on Software Process, Berlin, Germany, 1993, pp 75-93. 4. N. Belkhatir, W. L. Melo, Supporting Software Development Processes in Adele2, in the Computer Journal, vol 37, N°7, 1994, pp 621-628.. 5. K. Benali, N. Boudjlida, F. Charoy, J. C. Derniame, C. Godart, Ph. Griffiths, V. Gruhn, Ph. Jamart, D. Oldfield, F. Oquendo, Presentation of the ALF project, Proc. Int. Conf. on System Development Environments and Factories, 1989. 6. B. Boehm, A Spiral Model of Software Development and Enhancement, IEEE Computer 21, 5, 1988. 7. S. Brikemper, Formalisation of information systems Modelling, Ph. D. Thesis, University of Nijmegen, Thesis Publishers, Amsterdam, 1990. 8. J. Bubenko, C. Rolland, P. Loucopoulos, V. De Antonellis, Facilitationg Fuzzy to Formal Requirements Modelling, In the Proc. of the 1st ICRE Conf., Colorado Springs, USA, April, 1994
A Comprehensive View of Process Engineering
21
9. G. Cugola, E Di Nitto, A. Fuggetta, C. Ghezzi, A farmework for formalizing Inconsistencies and deviations in human centred systems, ACM Transactions on software engineering and methodology (TOSEM), Vol 5, N° 3, July 1996. 10. B. Curtis, M. Kellner, J. Over, A Field Study of the Software Design Process for Large Systems, Com. ACM, Vol. 31, No. 11, 1988. 11. B. Curtis, M. Kellner, J. Over, Process Modeling, Communications of ACM, vol 35 n°9, september 1992, pp 75-90. 12. De Antonellis V., Pernici B., Samarati P. (1991) F-ORM METHOD : A methodology for reusing specifications, in Object Oriented Approach in Information Systems, Van Assche F., Moulin B., Rolland C. (eds), North Holland, 1991 13. M. Dowson, Iteration in the Software Process, Proc 9th Int. Conf. on Software Engineering, 1988. 14. M. Dowson, Software Process Themes and Issues, IEEE 2nd Int. Conf. on the Software Process , pp 28-40, 1993. 15. M. Dowson, C. Fernstrom, Towards requirements for Enactement Mechanisms, Proc. of the th European Workshop on Software Process Technology, 1994 16. W. Emmerich, G. Junkermann, W Schafer, MERLIN : knowledge-based process modeling, Proc. of the First European Workshop on Software Process Modeling, Milan, Italy, 1991. 17. C. Fernström, L. Ohlsson, Integration Needs in Process Enacted Environments, Proc. 1st Int. Conf. on the Software Process, IEEE computer Society Press, October 1991. 18. Finkelstein A. , Kramer J. , Goedicke M. : ViewPoint Oriented Software Development; Proc. Conf Le Génie Logiciel et ses Applications, Toulouse, p 337351, 1990. 19. A. Finkelstein, J. Kramer, B. Nuseibeh (eds), Software Process Modelling and Technology, John Wiley (pub), 1994. 20. O. C. Z. Gotel, A. C. W. Finkelstein, An analysis of the requirements traceability problem, In Proc. Of Int. Conf. On Requirements engineering, ICRE’94. 21. M. Franckson, C. Peugeot, Specification of the Object and Process Modeling Language, ESF Report n° D122-OPML-1. 0, 1991. 22. Gamma E., Helm R., Johnson R., Vlissides J., Design patterns : Abstraction and Reuse of Object-Oriented Design, Proc. of the ECOOP'93 Conf., Sringer Verlag, 1993 23. Harmsen A.F., Brinkkemper J.N., Oei J.L.H.; Situational Method Engineering for information Systems Project Approaches, Int. IFIP WG8. 1 Conf. in CRIS series : Methods and associated Tools for the Information Systems Life Cycle (A-55), North Holland (Pub. ), 1994. 24. Harmsen F., Brinkkemper S., Design and implementation of a method base management system for situational CASE environment. Proc. 2nd APSEC Conf., IEEE Computer Society Press, pp 430-438, 1995 25. K. E. Huff, V. R. Lessor, A plan-based intelligent assistant that supports the software development process, Proc. of the 3rd Software Engineering Symposium on Practical Software Development Environments, Soft. Eng. Notes, 13, 5, 1989, pp97-106
22
Colette Rolland
26. Humphrey, W. S. : Managing the Software Process, Addison-Wesley, 1989. (verifier CMM) 27. Humphrey W. S, P. H Feiler, Software Process Development and Enactment : Concepts and Definitions, Tech. Report SEI-92-TR-4, SEI Institute, Pittsburgh, 1992 28. L. Jacherri, J. O. Larseon, R. Conradi, Sotware Process Modelling and Evolution in EPOS, in Proc. of the 4th Int. Conf. on Software Engineering and Knowledge Engineering (SEKE'92), Capri, Italy, 1992, pp574-589. 29. M. Jarke, J. Mylopoulos, J. W. Schmidt, Y. Vassiliou, DAIDA - An Environment for Evolving Information Systems; ACM Trans. on Information Systems, Vol. 10, No. 1, 1992. 30. M. Jarke, K. Pohl, Requirements Engineering: An Integrated View of Representation, Process and Domain, Proc. 4th European Software Conf., Springer Verlag, 1993 31. M. Jarke, K. Pohl, C. Rolland, J. R. Schmitt, Experienced-Based Method Evaluation and Improvement : A Process Modeling Approach, Int. IFIP WG8. 1 Conf. in CRIS series : Method and associated Tools for the Information Systems Life Cycle, North Holland (Pub. ), 1994. 32. Johnson R. E., Foote B., Designing reusable classes, Journal of Object-Oriented Programming, Vol 1, No3, 1988 33. G. E. Kaiser, N. S. Barghouti, P. H. Feiler, R. W. Schwanke, Database Support for Knowledge-Based Engineering Environments, IEEE Expert, 3(2), 1988, pp1832. 34. Kelly S., Lyyttinen K., Rossi M., Meta Edit+: A fully configurable, multi-user and multi-tool CASE and CAME environment, Proc. CAiSE'96 Conf., Springer Verlag, 1996 35. M. M. Lehman, Process models, process programming, Programming support, Proccedings of the 9th Int. Conf. on software engineering, Monterey, California, USA, 1987 36. J. Lonchamp, A structured Conceptual and Terminological Framework for Software Process Engineering, Proc. Int Conf. on Software Process, 1993 37. M. Lubars, C. Potts, C. Richter, A Review of the State of the Practice in Requirements Modeling, Proc. Int. Symposium on Requirements Engineering, 1993. 38. Merbeth G., Maestro II- das intergrierte CASE-system von Softlab, CASE systeme and Werkzeuge (Ed. H. Balzert) BI Wissenschaftsverlag, pp 319-336,1991 39. R. S Michalski, A Theory and Methodology of Inductive Learning, Atificial Intelligence, Vol 20, No 2, 1983 40. M. Oivo, V. R. Basili, representing software engineering model : the TAME goal oriented approach, IEEE Transactions on Software Engineering, Vol. 18 , N° 10, 1992. 41. T. W. Olle, J. Hagelstein, I. MacDonald, C. Rolland, F. Van Assche, A. A. Verrijn-Stuart, Information Systems Methodologies : A Framework for Understanding, Addison Wesley (Pub. ), 1988. 42. L. Osterweil, Software processes are software too; Proc. 9th Int. Conf. on Software Engineering, IEEE Computer Societ, Washington, DC, 1987, pp2-13
A Comprehensive View of Process Engineering
23
43. K. Pohl, Quality information systems : Repository for eveloving process models, Aachen Informatik, Beichte 92-37-RWTH, Aachen. 44. K. Pohl, The three dimensions of Requirements engineering. In Proc. of the 5th Int. Conf. on advanced Information Systems Engineering, pp. 275-292, Paris, France, June 1993. Springer-Verlag. 45. V. Plihon, C. Rolland, Modelling Ways-of-Working, Proc 7th Int. Conf. on Advanced Information Systems Engineering (CAISE), Springer Verlag, 1995. 46. C. Potts, A Generic Model for Representing Design Methods, Proc. 11th Int. Conf. on Software Engineering, 1989. 47. N. Prat, Using Machine learning techniques to Improve Information Systems Development Methods, 2nd AIS Americas Conf. on Information Systems, Phoenix, USA, 1996. 48. N. Prakash, Towards a formal definition of methods, in Requirements Engineering, Vol. 2 , N° 1, 1997. 49. Pree W., Design Patterns for Object-Oriented Software Development, Addison Wesley, 1995 50. R. Prieto-Diaz, P. Freeman, Classifying Software for reusability, IEEE Software, Vol. 4, N° 1, January 1987. 51. C. riesbeck, R. Schank, Inside Case-based Reasoning, Erlbaum(ed.), Northvale, New Jersey, USA, 1989 52. C. Rolland, Modeling the Requirements Engineering Process, Information Modelling and Knowledge Bases, IOS Press, 1993. 53. Rolland C., A Contextual Approach to modeling the Requirements Engineering Process, SEKE'94, 6th Int. Conf. on Software Engineering and Knowledge Engineering, Vilnius, Lithuania, 1994 54. C. Rolland, Modelling the evolution of artifacts, In Proc. of the first Int. Conf. on Requirements Engineering, April, 1994. 55. C. Rolland, M. Moreno, C. Souveyet, An approach for beginning ways of working, In Information System Journal, Vol. 20, N° 4, 1995. 56. Rolland C., Plihon V., Using generic chunks to generate process models fragments in Proc.of 2nd IEEE Int. Conf. on Requirements Engineering, ICRE'96, Colorado Spring, 1996 57. C. Rolland, N. Prakash, A proposal for context-specific method engineering, IFIP WG 8.1 Conf. on Method Engineering, Chapman and Hall, pp 191-208, 1996. 58. C. Rolland, A Primer For Method Engineering, In Actes du congrès Inforsid 97, Toulouse, France, June 1997. 59. C. Rolland, C. Ben Achour, C. Cauvet, J. Ralyté, A. Sutcliffe, N.A.M. Maiden, M. Jarke, P. Haumer, K. Pohl, Dubois, P. Heymans, A proposal for a scenario classification framework. To appear in Requirements Engineering Journal 3 :1, 1998. 60. T. Rose, M. Jarke, M. Gocek, C. Maltzahn, H. W. Nissen, A Decision-based Configuration Process Environment, IEEE Software Engineering Journal, 6, 3, 1991 61. Royce W. W. : Managing the Development of Large Software Systems; Proc. IEEE WESCON 08/1970
24
Colette Rolland
62. S. Si-Said, C. Rolland, G. Grosz, MENTOR :A Computer Aided Requirements Engineering Environment, in Proc. of CAiSE' 96, Crete, GREECE, May 1996. 63. S. Si Said, Guidance for requirements engineering processes. Proc. of the 8th Int. Conf. and Workshop on Database and Experts System Application DEXA’97, Toulouse, 1-5 September 1997. 64. K. Tominaga, T. Tokuda, Constraint-Centered Descriptions for Automated Tool Invocation, IEEE Asia-Pacific Software Engineering Conf. (APSEC), 1994, pp92101. 65. K. Van Slooten, B. Hodes, Characterising IS development project, IFIP WG 8.1 Conf. on Method Engineering, Chapman and Hall, pp 29-44, 1996. 66. Wilenski, Planning and Understanding, Addison Wesley (Pub. ), 1983. 67. Wirfs-Brock J., Johnson R., Surveying current research in Object-Oriented Design, Communications of ACM, Vol. 33, No9, 1990 68. P. H. Feiler, W. S. Humphrey, Software Process Development and Enactment: Concepts and Definitions, Proc. 2nd Int. Conf. on "Software Process", 1993.
A Comprehensive View of Process Engineering Colette Rolland University Paris-1 Sorbonne, 17, rue de la Sorbonne, 75005 Paris cedex 5, FRANCE email : rolland@univ-paris 1.fr
Abstract : The paper proposes a faceted framework to understand and classify issues in system development process engineering. The framework identifies four different but complementary view-points. Each view allows us to capture a particular aspect of process engineering. Inter-relationships between these aspects allow us to show the influence that one aspect has on another. In order to study, understand and classify a particular aspect of process engineering in its diversity we associate a set offacets with each aspect. The paper uses the framework to reuse questions, problems and research issues in the field.
1. INTRODUCTION Process engineering is considered today as a key issue by both the software engineering and information systems engineering communities. Recent interest in process engineering is part of the shift of focus from the product to the process view of systems development. Process engineering is a rather new research area. Consequently there is no consensus on, for example, what would be a good formalism to represent processes in, or, even, on what the final objectives of process engineering are Arm93 . However, there is already considerable evidence for believing that there shall be both, improved productivity of the software systems industry and improved systems quality, as a result of improved development processes Dow93, Arm93 and Jar94. Studies of software development practices Lub93, however, demonstrate that we know very little about the development process. Thus, to realise the promise of systems development processes, there is a great need Dow93 for a conceptual process engineering framework. In this paper we consider process engineering from four different, but complementary, view-points. Each view allows us to capture a particular aspect of process engineering. Inter-relationships between these aspects allow us to show the influence that one aspect has on another. In order to study, understand and classify a particular aspect of process engineering in its diversity we associate a set of facets with each aspect. For example, in the development view, where the concern is with the way in which process models are developed, it is possible to turn to (a) the facet called construction approach to understand how a process model can be constructed, (b) the construction technique facet to understand how it can be engineered, (c) the change support facet to see how flexible the process model is etc.. Facets have been proposed by Pri87 for classifying reusable components. They have also been used by Ro198 in requirements engineering for understanding and classifying scenario based approaches. When used in process engineering, a facet
provides a means for classification. For example, the coverage facet of the system world (see section 5 below) helps in classifying process models according to the underlying paradigm used: activity-oriented, product-oriented, decision-oriented or contextual. Each facet is measured by a set of relevant attributes. For instance, the description facet is measured by two attributes, the form and the notation attributes. Attributes have values which are defined in a domain. A domain may be a predefined type (INTEGER, BOOLEAN ...), an enumerated type (ENUM {x, y, z}), or a structured type (SET or TUPLE). We use the four worlds framework as a baseline and attach (a) a view of process engineering to each of its worlds and (b) a set of facets to each view. As a result, it is possible to identify and investigate four major view points of process engineering: what are processes, how are they represented, how can their representation be developed and used, and, finally, what does process engineering achieve. The multi-facet, multi-view approach adopted here makes it possible to look at process engineering in a comprehensive manner: -facets provide an in-depth description of each aspect of process engineering whereas aspects give a view of process engineering in all its diversity. - relationships between facets help in understanding the implications of one view on another.
2. THE FOUR-WORLDS FRAMEWORK The four worlds framework originally proposed for system engineering has proved its efficiency in enhancing the understanding of various engineering disciplines, information systems engineering [Jar92], requirements engineering [Jar93], and method engineering [Ro197]. It can also help in understanding the field of process engineering which consists of applying engineering approaches, techniques, and tools to the construction of process models.
Fig. I. The four worlds of process engineering In the original system engineering framework (Fig. 1.), the subject world contains knowledge of the domain about which the proposed IS has to provide information. It contains real-world objects which become the subject matter for system modelling.
The system world includes specifications at different levels of detail of what the system does. It holds the modelled entities, events, processes, etc. of the subject world as well as the mapping onto design specifications and implementations. The usage world describes the organisational environment of the information system, i.e. the activity of agents and how the system is used to achieve work, including the stakeholders who are system owners and users. The usage world deals with the intentional aspects of the IS to be built whereas the subject world refers to the domain it shall represent. The development world focuses on the entities and activities which arise as part of the engineering process itself. It contains the processes which create the information system i.e. processes which involve analysis and understanding of knowledge contained in the other worlds and representation of that knowledge. For our purposes, we identify the subject world as the world of processes. The system world deals with the representation of processes through process models. In the usage world we will investigate the reasons, the rationale for process engineering and relate the objectives of the users to the process models that can best meet these objectives. The development world deals with the process of constructing process models. This process is a meta-process in the sense that it supports the construction of processes, which in turn, will support the development of systems. The way the process might be supported by a tool environment is also a concern of this world. The paper uses the four worlds to present the state of art in process engineering and to raise questions, problems and research issues in the field. It comprises four sections, each of these relating to one of the world. This allows us to discuss in a focused manner the different concerns of process engineering : the definitions of processes, their representations, the way of developing these representations, and the rationale for using these representations. This is done in the subject, system, development, and usage worlds respectively.
3. THE SUBJECT WORLD Our Universe of Discourse is the world of processes. In this world, it is of interest to look at the notion of a process and its nature. A process is performed to produce a product. It has been described in the information systems area 01188 as the route to be followed to reach a product. This basic notion has been extended by Poh93 who looks upon a product as a point in threedimensional space comprising of the agreement, specification, and representation dimensions. Starting from some initial position in this space, the product moves through a succession of locations before a final position is reached. This final position corresponds to the desired product. The process then can be considered to be the route starting from the initial product position and going through the succession of intermediate positions till the fmal product position is reached.
The term process has been defined differently in different coverage (see section V below for the notion of coverage). In the activity-oriented coverage it is defined as a related set of activities conducted for the specific purpose of product definition. In Fei93 it is defined as "a set of partiaUy ordered steps intended to reach a goal" and a process step is itself an atomic action of a process that has no externally visible substructure. In the product-oriented coverage, a process is a series of activities that cause successive product transformations to reach the desired product. Fra91, Hum89 and Lon93 are three examples of definitions conforming to this view. In the decision-oriented coverage, a process is defmed as a set of related decisions conducted for the specific purpose of product definition. This view has been developed, for instance in IBIS Pot89, DAIDAJar92 and Ros91. Finally, in the coverage called context, a process is a sequence of contexts causing successive product transformations under the influence of a decision taken in a context Jar93. More intrinsically processes can be of different kinds. These various definitions reflect the multiple view points of the community about what is a process. However, these view points correspond to the various ways in which a process can be modelled and we will deal with in the system world. Strategic processes are those that investigate alternative ways of doing a thing and eventually, produce a plan for doing it. There are many ways of doing the same thing and the best way, the one most suited to the situation at hand has to be found. Such processes are creative and alternative generation and selection from an alternative are very critical activities here. Tactical processes are those which help in the achievement of a plan. As their name implies they are more concerned with the tactics to be adopted for actual plan achievement than with the development of a plan of achievement. Implementation processes are the lowest level processes. They are directly concerned with the details of the what and how of plan implementation. Thus, the subject world can be characterised by a facet having only one attribute called Nature defined as Nature: ENUM{strategic, tactical, implementation} As one can expect, we shall see below how the nature of the processes handled will influence the choice of a model adequate for their representation.
4. THE USAGE WORLD The usage world is where the goals of process use are established and, consequently, the range of facilities required for process performance are determined. The usage world can be viewed Dow93 as composed of three interacting domains : a process model domain, a process performance domain, and a process model enactment domain (Fig. 2.).
I Process ModelDomain ~1 ~Process Performance Domain 1 Model . I~ (~ ~ & agents, Fragments
\ Process Improvement, Capitalisation of Experience
t~
\ Enactement Creation
-,,
r ~ l
|-- ~ Activities
.,,+" Guidance Monitoring/Controling/
/ Feedback
/
t ProcessMode, Enac,eme. J Domai n Enactement Mechanism
Fig. 2. Process domains The process model domain contains process models. A process model describes the common properties of a class of processes having the same nature. The process performance domain deals with the actual activities conducted by human agents and machines, in the course of a project. Some will be executed by software tools; others will consist of human thinking, writing, exchanging ideas, and taking decisions through formal and informal interactions between members of the project team. All these activities must be supported by the process model. The process model enactment domain is concerned with the features needed to support process performance governed by the process model. These features support, guide, or enforce performance of the process in a way consistent with the process model. The three domains interact with each other in different ways. Firstly, the process model influences the way in which the process is performed. Actual performance then corresponds to some extent to the model of how it should be performed. Secondly, the course of enactment may need to be contingent on events arising from actual process performance. Therefore, the actual process will be different from the theoretical instantiation of the process model. This leads to the idea of feedback from process trace to process model, thereby allowing its improvement. This leads to a view of the usage world as imposing strong requirements on the way processes will be performed, the nature of process models used and the way in which these process models will be developed and changed. The purpose assigned to the process model has to be determined by the usage world. This is captured below in the facet, Purpose. Since the way processes are performed changes with time, it is the duty of the organisation to define their process management policy. This is captured in the facet, Process Management Policy.
4.1 PURPOSES
A synthesis of proposals from the software engineering field Lon93, Cur92, the information system community Bri90, Pra97, Ro196a, and the system design community Ram92, Pot89, show three main aims of process models: - descriptive, to record trace what actually happens during a process, - prescriptive, to define desired processes and how they should/could/might be performed, - explanatory, to provide explanations about the rationale of processes. A descriptive purpose takes the point of view of an external observer who looks at the
way a process has been performed and determines the improvements that have to be made to make it perform more effectively or efficiently. The prescriptive purpose lays down rules, guidelines, and behaviour patterns which, if followed, would lead to the desired process performance. The prescriptive purpose lies in a range from strict enforcement to flexible guidance. In the former the performance of the process must follow the prescription whereas in the latter the prescription is such that it can accommodate a large number of ways in which the process can proceed. Guidance shifts the emphasis away from task performance to goal achievement. Therefore, there can be two types of guidance, point and flow ISis97. Point guidance provides help in the achievement of a given goal whereas flow guidance helps in identifying the next goal in order for the process to proceed. The explanatory purpose is important in those processes where several possible
courses of action are open and each of these has to be explored and evaluated based on rational arguments. Such traces establish an explicit link between processes and the requirements that they are to fulfil. The descriptive and explanatory purposes have been accorded a lot of attention in the recent past. This is because of the need to keep track of process knowledge and to support change Got94, Ram92. To take this to the extreme, it is difficult to visualise any process, strategic, tactical, or implementation (see Subject World), without a descriptive and/or explanatory purpose behind them. Specifically, if prescription is to be provided to strategic processes, then flexible guidance is clearly more appropriate than process enforcement. This is because strategic processes are often creative and require human co-operation. This makes most software process models inappropriate for strategic processes because Fin94 their basic property is enforcement of constraints (prescriptions and even proscriptions). However, in tactical or implementation processes of the Subject World that follow plans relatively more strictly and which are less creative and mercurial, varying shades of process enforcement ranging from mechanical enforcement with limited guidance to complete automation may be found useful.
A process engineering approach can be classified according to the role it aims to play in the facet called Purpose which has the three following attributes : Prescriptive: BOOLEAN Descriptive: BOOLEAN Explanatory: BOOLEAN 4.2 PROCESS MANAGEMENTPOLICY Processes change with time and so do the process models underlying them. Thus, new processes and models may have to be built and existing ones improved. There is need to have a well-defmed organisational policy to handle this change. This policy can either accept change continuously as it occurs or accept it as one-shot, radical change. Radical change applies in situations where organisations need to define a process management policy from scratch. The former applies when need is felt to harmonise heterogeneous process practices or when a bottom-up approach is systematically applied to move up in the levels of maturity in the CMM Hum89 framework.. Strategic processes are highly unstable. The process proceeds by analogy with other similar processes and reuses experience and knowledge of their stakeholders. This reuse is continuous and operates so long as the process lasts. It is today implicitly done by individual human agents performing the process but, perhaps, in future, it shall be necessary to have reuse as a process management policy of the organisation. However, it remains to be conclusively shown that process practice reuse is cost effective in an organisational setting. The foregoing is captured by the two attributes change and reuse of the Process Management Policy facet Change: ENUM{continuous, radical} Reuse: BOOLEAN
5. THE SYSTEM WORLD If the subject world is the world of processes then the system world is the one of their representations. The interest in this world is in a) what is to be represented b) at what level of abstraction c) how is it represented d) what properties should the representation have.
The facet contents, of the system world deals with (a), the abstractionfacet deals with (b), the descriptionfacet deals with (c), and finally, the modularizationfacet captures the properties of the representation. We develop each of these below.
5.1 ABSTRACTION Processes of the same nature are classified together into a process model. Thus, a process model is a description of a process at the type level. Since the process model is at the type level, a process is an instantiationl of it. The same process model is used repeatedly for the development of many applications and thus, has many instantiations. As stated in section 4, one possible use of a process model is to prescribe "how things must/should/could be done" in contrast to the process itself which is really what happens. A process model is more or less a rough anticipation of what the process will look like. What the process shall, indeed, be will, however, be determined during actual system development.
A process meta-model is a description at the type level of a process model. A process model is, thus, an instantiation of a process meta-model. Obviously, a meta-model can be instantiated several times in order to defme various process models. A process meta-model is at the meta-type level with respect to a process. It plays a role similar to a theory in the Theory of Plans Wi183 from which plans can be generated (the process models) and executed (the processes).
The abstraction facet captures the levels at which the model is expressed and the corresponding attribute takes on values from the enumerated domain { type, meta-
type}. The well known models like the waterfall Roy70 and spiral models Boe88 are the type level whereas the Nature process theory Ro195 is at the meta-type level.
5.2 CONTENTS The concern of this facet is with the contents of the process model/meta-model. These contents are determined by the system of concepts in terms of which processes are represented and by the granularity of these representations. These two aspects are dealt with by the coverage and granularity attributes respectively. 5.2.1
Coverage
According to Dowson Dow88, models: - activity-oriented, - product-oriented, and - decision-oriented.
process models can be classified into three groups of
A. Finkelstein in (Fin94) points out the various meaning of the widely used term "instantiation" in the software engineering community. We relate here to the classical idea of creating instances from a type/class definition
Since this classification was made, a new group called the contextual model has also emerged.
Activity-oriented The activity-oriented models concentrate on the activities performed in producing a product and their ordering. These process models are sequential in nature and adopt the Cartesian, functional decomposition approach. They provide a frame for manual management of projects developed in a linear fashion. The first widely used process model, the Waterfall model Roy70, falls into this category. Its widespread acceptance has led to life-cycle descriptions being most often treated as linear sequences where crucial aspects of the process such as feedback loops and iteration are not represented Boe88, Cur88 and Cur92. These models are well suited to model implementation processes. The strong emphasis on an activity incurs some risks of neglecting the influence of product structure on the process. Further, they are unable to support flexible prescriptive guidance but only process model enforcement. The linear view of activity decomposition seems inadequate to model creative processes because it is not possible to consider all contingencies. Activity-oriented representations cannot incorporate the rationale underlying the process and therefore do not permit reasoning about engineering choices based on existing conditions. It is unrealistic to plan what will happen in such a process in an entirely sequential manner. Finally, the linear view is inadequate for process models which have to support backtracking, reuse of previous designs and parallel engineering.
Product-oriented Product-oriented process models, in a manner similar to activity-oriented ones, are centred around the notion of activity but, additionally, link activities to their output : the product. The ViewPoints model Fin90 and the process model proposed in the European Software Factory (ESF) project Fra91 belong to this category. Product-oriented models couple the product state to the activities that generate this state. They visualise the process as a state transition diagram. Since product-oriented models adopt the notion of an activity, they suffer from the same difficulties as the activity-oriented models considered above. However, due to their product-activity coupling they are useful for tracing the transformations performed and their resulting products. However for strategic processes it is difficult, if not impossible, to write down a realistic state-transition diagram.
Decision-oriented The successive transformations of the product caused by a process are looked upon, in decision-oriented models, as consequences of decisions. The process models of the DAIDA project Jar92, Pot89 and Ram92 fall into this category. These models emphasise the concept of an "Intention" at the expense of an activity.
10
Decision-oriented models can be used for both, strategic as well as tactical processes. The strength of the decision-oriented approach is its ability to cover more aspects of a process than can be done by the two other kinds. Decision-oriented models are not only able to explain how the process proceeds but also why it proceeds. Therefore, decision-oriented process models are particularly well suited to strategic processes, for supporting explanatory tracing and prescriptive guidance. This is because of their ability to (a) guide the decision making process (b) help in reasoning about the rationale behind decisions,(c) support the deliberation underlying the decision process itself and (d) keep a trace of the happenings of a process and their rationale. Contextual Models Contextual models as found in the Nature process theory Bub94, and in the F3 project Ro194b look upon each process as being in a subjectively perceived situation upon which is looked upon with some specific intention. The work to be done next depends on both the situation and the intention i.e. it depends on the context existing at this moment. Contextual process models strongly couple the context of a decision to the decision itself. It makes the notion of a context, the coupling of a situation and the decision, central to process modelling. Decisions are applied to the situation in which the process currently is, in order to change that situation to the desired new one. In this respect, the contextual approach has some similarity with the planning paradigm that has emerged from Artificial Intelligence and with projects based on the planning paradigm such as GRAPPLE Huf89. Since the contextual models adopt the notion of a decision, all the properties of decision-oriented models mentioned earlier are applicable to them. Further, due to the strong relationship between the situation and the decision, only those decisions which are appropriate in the situation at hand are of interest. This helps in focusing guidance, tracing and explanation to specific process situations. Process models can be classified within the facet Contents, by giving values to the attribute, coverage, Coverage: ENUM{activity, product, decision, context}
5.2.2 Granularity Most traditional process models are large-grained descriptions of the product lifecycle. On the other hand, there are very fine-grained models. For example specifying that after a source code file is edited, it should be recompiled Kai88. Recently, hybrid formalisms that use different notations for large-grain and small-grain aspects of process such as PROCESS WEAVER Fer91, have been developed.
11 The nature of granularity needed is dependent on the situation at hand. Granularity affects the kind of guidance, explanation and trace that can be provided. High granularity limits these to a rather coarse level of detail whereas free granularity provides more detailed capability. Process models should, ideally, provide a wide range of granularity. This shall allow a movement from large grains to fine grains along a continuum. Therefore, the granularity attribute takes on values from SET(ENUM{large, fine, variable}). 5.3 THE DESCRIPTION FACET
The description facet is concerned with the form of the process representation and the level of formality of the language used to describe the process model. These correspond to the form and notation attributes of this facet. 5.3.1 Form
The form attribute is concerned with style of the process representation. There are three identified forms, scripts, programs, and hypertext. Osterweil lOst87 has proposed the view that software process models should take the form of a program as different from process scripts. Process scripts are interactively used by humans as against process programs which are enacted by a machine Leh87. They support non determinism whereas process programs can, at best, support process deviation under pre-defined constraints Cug96. The hypertext style of process representation is a network of links between the different aspects of a process, such as product parts, decisions, arguments, issues, etc. A relationship can be established between form and the purpose facets of the Usage World. Scripts and programs are two styles which may be applicable to prescriptive purposes whereas hypertext is well suited to descriptive and explanatory purposes. Strict enforcement of the prescriptive purpose can clearly be represented in process programs whereas flexible guidance requires the process model to be represented in process scripts. Descriptive and explanatory purposes require the establishment of relationships between different elements of a process trace. These relationships are well articulated as hypertext links. The form attribute of the description facet takes on values from ENUM{script, program, hypertext} 5.3.2 Notation
Process models underlying information systems practice have traditionally used informal notations such as natural languages or diagrams with informal semantics. On the other hand, in software engineering, more formal software process models (see Arm93, Cur92, Fin94 for an overview) have been used. This formality relates to
12 underlying programming languages : Smalltalk for E3 Fin94, various Prolog dialects for EPOS Jac92, Oikos lAmb91, and PEACE Fin94, PS-Algol for PWI Fin94. A formal notation is required to support the verification of the expected properties of the process model and validation of the process model using for instance, simulation or enactment techniques. The use of informal notations has made it difficult for process models to be followed systematically. Formal or semi-formal notations make these efforts considerably more effective. Formal notations are necessary for providing automatic enactment support.
The notation attribute helps classifying process models by one of the three values of the following enumeration: Notation: ENUM{forrnal, semi-formal, informal} 5.4 MODULARIZATION
Early processes were monolithic. However, there is a shift towards modular process structure in this decade. We introduce a Boolean valued attribute called Presence in the modularization facet to distinguish between monolithic and modular methods.
One proposal for modularization Har94 is that of fragments. A fragment can be either a product fragment or a process fragment. The drawback of the fragment based approach is the over-emphasis on the product fragment resulting in under developed meta-process modelling. The proposal of Ro193, Ro194a, is to tightly couple the product and process aspects of processes into contexts. A Context is a couple <situation, decision>, where the decision part represents the choice an IS developer can make at a moment in the engineering process and the situation is defined as the part of the product it makes sense to make a decision on. Process modules can be looked upon according to two other perspectives : abstraction and aggregation. Rolland Ro195 has defined aggregates called process chunks as hierarchies of contexts. A chunk prescribes the way to proceed in the situation identified by the context at the root of its context hierarchy. This allows the decision of the root context to be taken in this situation. IVan96 proposes two kinds of aggregated modules called route map an d fragments respectively. A route map refers to strategies such as delivery strategies, developmental strategies, realisation strategies etc., activities and products concerning system development as well as project management. The fragment is a coherent part of a process for system development or project management. Fragments may be linked to a route map which may establish a complete project approach.
13 Abstraction is used to capture generic laws goveming the construction of different but similar process modules. Generic process modules can take the form Ro196a of framework or pattern. A framework models the commonality between modules of different process models but for the same type of application. A pattern models a common behaviour in process construction. It is generic in the sense that it is used every time a process model is constructed. Both terms have been chosen by analogy with reuse approaches in the object oriented area. Patterns are there defined as solutions to generic problems which arise in many applications Gam93, Pre95 whereas a framework is application domain dependent Wir90, Joh88.
Classification along the modularizationfacet comes down to giving values to the two following attributes: Presence: BOOLEAN Nature: SET( ENUM{primitive, aggregate, generic}
6. THE DEVELOPMENT WORLD The development world deals with two issues - the process ofconstructingprocess models, and - enactment of processes. The process of constructing process models is a meta-process, it is the process behind the process used to construct processes for building information systems products. The development world deals with meta-processes so as to improve process models and to make them evolve. The second issue is that of process enactment. The development world is also concerned with the way in which process models can be constructed and process enactment support provided. That is, the tool environment needed to support process performance is also a concern of this world. Thus, the facets of interest in this world are construction approach, construction technique, enactment support, and change support.
6.1 CONSTRUCTIONAPPROACH In a manner analogous to that of Harmsen Har94 one can organise construction approaches in a spectrum ranging from 'low' flexibility to 'high'. At the 'low' end of this spectrum are rigid approach whereas at the 'high' end is modular approach. Rigid approaches lead to process models that are completely pre-defmed and leave little scope for adapting them to the situation at hand. On the other hand, contingency approaches allow the modification and augmentation of models to make them fit to a given situation.
14 There are at least two ways by which contingency approaches can be realised. The first one is the production of contingency process models that is, situation-specific models for certain types of organisational settings. This presents process engineering as the selection of a model within a panel of contingency process models. In the second one process engineering is used to support the selection and the assembly of process components to construct process models 'on-the-fly'. The foregoing suggests that construction approach should be classified as : Construction approach: ENUM{contingency, on-the-fly, rigid} The construction approach adopted in the development world has a strong impact on the modularization facet and granularity attribute of the system world. Whereas the rigid approach can be associated to monolithic models, contingency and on-the-fly approaches require modular process models. The contingency approach is well suited to support capitalisation of 'good practice' into process chunks in a stable environment. Instead 'on-the fly' approaches are adapted to the dynamic recognition and use of chunks and patterns. 6.2 CONSTRUCTIONTECHNIQUE Within the broad construction approach adopted for constructing process models, a number of techniques for construction are available. Construction techniques used in the information systems area have developed independently of those in software engineering. In information systems, construction techniques exploit the notion of a meta-model and the two principal techniques used are those of instantiation and assembly. In software engineering the main construction technique used today is language-based. However, early techniques in both, information systems and software engineering were based on the experience of process engineers and were, therefore, ad-hoc in nature. We comment the attributes values in turn. 6.2.1 Instantiation
Given that new process models shall be constructed very often, the question is how we can increase the productivity of process engineers and improve the quality of the models they produce. One way of doing this is to identify the common, generic features of process models and represent them in a system of concepts. Such a representation has the potential to 'generate' all process models that share these features. This potential is realised when a generation technique is defined whose application results in the desired process model. Thus, there are two key issues here - the identification of the system of generic concepts - the instantiation technique. The first of these is resolved by the definition of a process meta-model whereas the second issue is resolved by deriving process models from this process meta-model through instantiation. A number of advantages flows from this: 1) The exploitation of the meta-model helps us to define a wide range of process models. 2) It makes the activity of defining process models systematic and versatile.
15 3) It forces us to look for and introduce, in the process meta-model, generic solutions to problems and this makes the derived process models inherit the solution characteristics. Under the instantiation approach, the crucial issue in process modelling is no longer the process model but the process meta-model. This means that the onus of providing a process model with the required characteristics shifts from the process model developer to the process meta-model developer. The instantiation technique has been used, for example, in NATURE Ro193, Ro194, Ro194a, Ro196a. The process engineer must defme the instances of contexts and relationships that comprise the process model of interest. It has been utilised to build the repository of Computer Aided method Engineering environments Ke196, Har95, Mer91, Sis96.
6.2.2 Language The software engineering community has used different languages for expressing process models like Smalltalk for E3 Fin94, various Prolog dialects for EPOS Jac92, Oikos Amb91, and PEACE Fin94, PS-Algol for PWI Fin94. Different computational paradigms have also been used, for example, Petri nets in EPOS Jac92 and SPADE Ban93, rule based paradigm in MERLIN Emm91, ALF Ben89, Marvel Kai88, EPOS Jac92, and triggers in ADELE Be189 and MVP-L Fin94. There is a relationship between the construction technique and the form facet in the system world. Indeed, languages are typically related to process programs whereas instantiation techniques have been used to construct process scripts.
6.2.3 Assembly The assembly technique relies on the availability of process components in a process repository. Assuming that process components exist in a process repository, the question now is "how to deliver the relevant process components to the user?" The process community has been looking at this question in two ways : first, by promoting a global analysis of the project on hand based on contingency criteria and, secondly, by associating descriptors to components in order to ease the retrieval of components meeting the requirements of the user. Therefore in the former the project situation is at a very global level whereas in the latter the descriptors of process components support local matching with the situation at hand. Van96 is an example of the first approach. This approach has been tried out in nine non-standard projects of the systems development department of a bank organisation. The second approach Ro196b uses the notion of descriptor IDEA91 as a means to describe process chunks. It has been tried out to construct information systems methods Pii95 in NATURE and repository of scenario based approaches accessible on Interact in the CREWS project Ro198.
16
For the assembly technique to be successful, it is necessary that process models are modular. If the assembly technique is combined with the instantiation technique then the meta-model must itself be modular. 6.2.4
Ad-Hoe
Traditional process models are expressions of the experiences of their developers. Since this experience is not formalised and is, consequently, not available as a fund of knowledge, it can be said that these process models are the result of an ad-hoc construction technique. This has two major consequences : it is not possible to know how these process models were generated, and they become dependent on the domain of experience. If process models are to be domain independent and if they are to be rapidly generable and modifiable, then we need to go away from experience based process model construction. Clearly, generation and modifiability relate to the process management policy adopted (see Usage World). Instantiation and assembly, by promoting modularization, facilitate the capitalisation of good practice and the improvement of given process models. The construction technique facet is defined as follows: Construction technique: SET(ENUM{instantiation, language, assembly, ad-
hoc}) 6.3 ENACTMENTSUPPORT
Enactment mechanisms have been mainly implemented by the software engineering community as the core of Process Centred Software environments. An enactment mechanism determines and controls the interactions between the agents performing the process so as to trace, guide, and enforce performance of the process in a way consistent with the process model. Considerable efibrt has been put in to provide automated execution support, automated monitoring and enforcement of software processes in process centred software environments. The reader will find in Fin94 a detailed presentation of ten projects in the field as well as the results of a common assessment exercise performed by the leaders of these projects. Most process centred software environments Jac92, Be194, Ban93, Kai88 are in fact used to describe the activity of tools and to allow automatic invocation of tools Tom94. Existing environments guide software engineers in the selection of the right suite of tools but they do not guide the engineering activities themselves. On the contrary, some attempts have been made in the information systems community for implementing enactment mechanisms that focus on guiding engineering activities ISis96 Whereas the foregoing deals with supporting the performance of application processes, there is also need to support the process of constructing process models, the meta-process. Just as other processes are represented by process models, the meta-
17 process shall have its own model, the meta-process model. Again, the meta-process itself needs to be performed in accordance with the meta-process model and this means that enactment support has to be provided for the performance of the metaprocess. It is possible to build two separate enactment environments for dealing with process and meta-process enactment respectively. However, if the meta-process is treated as just another process then it is possible to use the same enactment mechanism to support both, the process and the meta-process. In fact this has been demonstrated in the Mentor environmentSis96. Enactment mechanisms must support processes that take the form (see System World) of scripts, programs, or hypertext. When a process model is a script then the enactment mechanism provides high flexibility so as to enable human agent intervention during process performance. This intervention would be supported by guidance mechanisms which may either, proactively provide suggestions on possible decisions that could be taken or may support requests for help. In terms of the Usage World, for models which are process programs, the enactment mechanism behaves like the run-time support of programming languages. Process program enactment is program execution whereas process script enactment is model interpretation. Finally, when models are of the hypertext form then the enactment mechanism offers facilities to create links between process decisions, their rationale, and the resulting product. Since the meta-process is a process, it is possible to extend the foregoing remarks to it as well. However, as it is unlikely to completely plan out the meta-process, it would be the case that meta-process models are not treated as process programs but as process scripts only. This facet has two Boolean values attributes Process support: BOOLEAN Meta-process support: BOOLEAN 6.4
CHANGE SUPPORT
The traditional practice is that if a process model does not meet requirements of the users then a new one is built. This practice causes loss of experimental knowledge which could have been used to change process models. The development world must therefore provide support for process model change. There are two different ways in which this can happen (a) process model change takes place even as the process proceeds: the process model can be adapted to specific requirements as these emerge, (b) the process model may need to be revised and improved at the end of the project: this is to benefit from the experience gained in process model use. The former is referred to as dynamic process change Dow94 and the latter as process improvement Lon93.
~8
Different positions are taken in the software process engineering community concerning the need for dynamic changes. On the one hand, people claim that this is an essential requirement and some software process centred environments EPOS Jac92, E3 Fin94, SPADE Ban93, ADELE Be189 try to provide solutions for it Fin94. On the other hand, it can be argued that a prescriptive approach to process modelling is at odds with dynamic process evolution Hum92. The notion of fitness of the process has been defined in Hum92 as the degree to which the agents performing the process can faithfully follow the process steps it specifies. When this fitness is low then process change occurs. Thus, process model change is an indication of lack of flexibility of the model. Recent process models include the concept of process deviation and therefore control the deviation of process performance from that prescribed in the process model. There are only initial experiments in improving the process model by experiencebased learning, as suggested in the literature Hum89, Oiv92, Poh92. They suggest two ways of process improvement, by inductive or deductive learning. Inductive learning is based on the analysis of process deviations that are recorded in process traces. Induction improvement can be performed by a human agent who, on his own, decides the change that is needed. The agent can be supported by generalisation rules Mic83 that can be part of a tool based inductive learning environment Pra96. In order to do inductive improvement, there must exist a mapping between elements of the trace and the concepts of the process model. Deductive learning exploits Case-based reasoning. Thus, it solves new problems by adapting solutions that have been utilised to resolve past problems Rie89. Case based reasoning when applied to process performance calls for the existence of a repository of cases. Deductive process improvement aims at adding new cases in the repository by examining process performances. Deductive learning corresponds to the retaining phase of the Case based reasoning cycle which traditionally consists of the four phases (a) retrieve, (b) reuse, (c) revise, and (d) retain. Dynamic process change and process improvement are the two techniques that the Development World can offer to support the process management policies set in the Usage world. Deductive process improvement is appropriate when an organisation wants to promote the reuse of good practice in performing processes. Clearly, a process model supporting aggregates (see Modularization facet in System World). shall be well suited to provide these as cases to be stored in the repository. Inductive improvement is well suited to situations where process models are used repeatedly and can continuously be improved by learning from the deviation of actual performances. A modular structure of process models helps in relating the observed deviations to specific, localised parts of the process model components and therefore facilitate inductive improvement.
19
The chage support attribute takes one or several values among the following enumerated domain : Change support: SET(ENUM {dynamic process change, process improvement})
7. CONCLUDING REMARKS The subject and the usage worlds constitute the environment within which the technical view of process engineering contained in the system and development worlds lies. This embedding is the source of the inter-relationships between the facets of the four views discussed in this paper. The nature of processes and the purpose imposed by the usage world on the process engineering solution determine, to a large extent, the type of contents and description of the process models/meta-models. The choice of a particular type of content and description based on the nature of the processes guarantees that the semantics of these processes are well captured in process models/meta-models. On the other hand, selection of a content and description to meet the purpose expressed by the usage world guarantees that the process model/meta-model shall fulfil the requirements of the process stakeholders. In fact, we suggest that selection based on purpose should have primacy over that based on the nature of the process. This conclusion can be drawn by analogy with ISE approaches where it has been recognised that user needs are better met by understanding usage goals and not merely by using good semantic conceptual model. In fact, the usage world affects the system world through both, the purpose as well as the process management policy. Of the three purposes, the explanatory and the descriptive have been incorporated in process models that provide design rationale and process traceability respectively. However, the issue of providing prescriptive guidance is still open. The process management policy affects the abstraction facet as it introduces the need for abstracting process models into process meta-models. Before building meta-models which reflect old models but on a new level of abstraction, one should question the old ones. The goal of meta-modelling is not only to operationalise current process models but also to correct the general oversights and limitations of these. In a similar manner, we believe that the technical solution in the development world has to be chosen according to the purpose and the process management policy decided in the usage world. The influence of the former is clearly on the choice of the enactment mechanisms. The implication of the latter is more diverse. The policy recognises the need for managing processes in-the-large and their evolution in time. The former sets the requirements of an automated enactment support and its extension to the meta-process. The latter determines the choice of the change support and of the construction approach. There are two implications of this, organisational and technical. The absence today of organisation-wide process policies raises the need for organisations to understand the crucial role played by these policies and to
20 define them. Such policies would, for example, encourage capitalisation of good practice, learning from experience, and development of a reuse culture. The capitalisation policy raises the technical question of how good practices can be recognised, organised and reused. Such knowledge should be available in the form of chunks for later assembly. This implies the modularization of process models/metamodels. The modular approach represents a shift in the way of thinking about process representations and is likely to emerge as a major research issue in the future. The reuse culture raises the question of the genericity of process representations. Perhaps, what is needed is a corpus of both, generic process and generic meta-process knowledge in a modular form. A suggested research is therefore, the development of a domain analysis approach to identify common properties of (a) different process models and (b) different meta-process models. The evolution of process models calls for the establishment of technical enactment support for the meta-process. So far, work has concentrated on developing and experimenting with process enactment mechanisms. The research issue here is of making these mechanisms generic enough to handle both process and meta-process enactment. This paper has argued that process engineering should be usage driven. The acceptance of process engineering in organisations is, however, not entirely determined by the functionality that is needed but also by other non-functional factors such as usability, availability, security etc. This aspect of process engineering has to be addressed by the research community more vigorously.
8. REFERENCES lAmb91 : V; Ambriola, M. L. Jaccheri, Definition and Enactment of Oikos software entities, Proc. of the First European Workshop on Software Process Modeling, Milan, Italy, 1991 Arm93 P. Armenise, S. Bandinelli, C. Ghezzi, A. Morzenti, A survey and assessment of software process representation formalisms Int. Journal of Software Engineering and Knowledge Engineering, Vol. 3, No. 3, 1993. Ban93 S. Bandinelli, A. Fugetta, S. Grigoli, Process Modelling in the large with SLANG, Proc. of the 2nd Int. Conf. on Software Process, Berlin, Germany, 1993, pp 75-93. Be194 N. Belkhatir, W. L. Melo, Supporting Software Development Processes in Adele2, in the Computer Journal, vol 37, N~ 1994, pp 621-628.. Ben89 K. Benali, N. Boudjlida, F. Charoy, J. C. Demiame, C. Godart, Ph. Griffiths, V. Gruhn, Ph. Jamart, D. Oldfield, F. Oquendo, Presentation of the ALF project, Proc. Int. Conf. on System Development Environments and Factories, 1989. Boe88 B. Boehm, A Spiral Model of Software Development and Enhancement, IEEE Computer 21, 5, 1988.
21 Boeh76 B. Boehm, Software Engineering, IEEE Transactions on Computers, Vol. C-25, No. 12, 1976. Bri90 S. Brikemper, Formalisation of information systems Modelling, P h . D . Thesis, University of Nijmegen, Thesis Publishers, Amsterdam, 1990. Bub94 J. Bubenko, C. Rolland, P. Loucopoulos, V. De Antonellis, Facilitationg Fuzzy to Formal Requirements Modelling, In the Proc. of the 1st ICRE Conf., Colorado Springs, USA, April, 1994 Cug96 G. Cugola, E Di Nitro, A. Fuggetta, C. Ghezzi, A farmework for formalizing Inconsistencies and deviations in human centred systems, ACM Transactions on software engineering and methodology (TOSEM), Vol 5, N ~ 3, July 1996. Cur88 B. Curtis, M. Kellner, J. Over, A Field Study of the Software Design Process for Large Systems, Com. ACM, Vol. 31, No. 11, 1988. Cur92 B. Curtis, M. Kellner, J. Over, Process Modeling, Communications of ACM, vol 35 n~ september 1992, pp 75-90. IDEA91 De Antonellis V., Pemici B., Samarati P. (1991) F-ORM METHOD : A methodology for reusing specifications, in Object Oriented Approach in Information Systems, Van Assche F., Moulin B., Rolland C. (eds), North Holland, 1991 Dow88 M. Dowson, Iteration in the Software Process, Proc 9th Int. Conf. on Software Engineering, 1988. Dow93 M. Dowson, Software Process Themes and Issues, IEEE 2rid Int. Conf. on the Software Process, pp 28-40, 1993. Dow94 M. Dowson, C. Femstrom, Towards requirements for Enactement Mechanisms, Proc. of the th European Workshop on Software Process Technology, 1994 Emm91 : W. Emmerich, G. Junkermann, W Schafer, MERLIN ." knowledge-based process modeling, Proc. of the First European Workshop on Software Process Modeling, Milan, Italy, 1991. Fer91 C. Fernstr6m, L. Ohlsson, Integration Needs in Process Enacted Environments, Proc. 1st Int. Conf. on the Software Process, IEEE computer Society Press, October 1991. Fin90 : Finkelstein A. , Kramer J. , Goedicke M. : ViewPoint Oriented Software Development; Proc. Conf Le G6nie Logiciel et ses Applications, Toulouse, p 337351, 1990. Fin94 A. Finkelstein, J. Kramer, B. Nuseibeh (eds), Software Process Modelling and Technology, John Wiley (pub), 1994. Got94 O. C. Z. Gotel, A. C. W. Finkelstein, An analysis of the requirements traceability problem, In Proc. Of Int. Conf. On Requirements engineering, ICRE'94. Fra91 M. Franckson, C. Peugeot, Specification of the Object and Process Modeling Language, ESF Report n ~ D122-OPML-1.0, 1991. Gain93 Gamma E., Helm R., Johnson R., Vlissides J., Design patterns : Abstraction and Reuse of Object-Oriented Design, Proc. of the ECOOP'93 Conf., Sringer Verlag, 1993 Har94 : Harmsen A.F., Brinkkemper J.N., Oei J.L.H.; Situational Method Engineering for information Systems Project Approaches, Int. IFIP WG8. 1 Conf. in
22 CRIS series : Methods and associated Tools for the Information Systems Life Cycle (A-55), North Holland (Pub.), 1994. Har95 Harmsen F., Brinkkemper S., Design and implementation of a method base management system for situational CASE environment. Proc. 2nd APSEC Conf., IEEE Computer Society Press, pp 430-438, 1995 Huf89 : K. E. Huff, V. R. Lessor, A plan-based intelligent assistant that supports the software development process, Proc. of the 3rd Software Engineering Symposium on Practical Software Development Environments, Soft. Eng. Notes, 13, 5, 1989, pp97-106 Hum89 Humphrey, W. S. : Managing the Software Process, Addison-Wesley, 1989. (verifier CMM) Hum92 Humphrey W. S, P. H Feiler, Software Process Development and Enactment : Concepts and Definitions, Tech. Report SEI-92-TR-4, SEI Institute, Pittsburgh, 1992 Jac92 L. Jacherri, J. O. Larseon, R. Couradi, Sotware Process Modelling and Evolution in EPOS, in Proc. of the 4th Int. Conf. on Software Engineering and Knowledge Engineering (SEKE'92), Capri, Italy, 1992, pp574-589. Jar92 M. Jarke, J. Mylopoulos, J. W. Schmidt, Y. Vassiliou, DAIDA - An Environment for Evolving Information Systems; ACM Trans. on Information Systems, Vol. 10, No. 1, 1992. Jar93 M. Jarke, K. Pohl, Requirements Engineering: An Integrated View of Representation, Process and Domain, Proc. 4th European Software Conf., Springer Verlag, 1993 Jar94 M. Jarke, K. Pohl, C. Rolland, J. R. Schmitt, Experienced-Based Method Evaluation and Improvement : A Process Modeling Approach, Int. IFIP WG8. 1 Conf. in CRIS series : Method and associated Tools for the Information Systems Life Cycle, North Holland (Pub.), 1994. Joh88 Johnson R. E., Foote B., Designing reusable classes, Journal of ObjectOriented Programming, Vol 1, No3, 1988 Kai88 G. E. Kaiser, N. S. Barghouti, P. H. Feiler, R. W. Schwanke, Database Support for Knowledge-Based Engineering Environments, IEEE Expert, 3(2), 1988, pp18-32. Ke196 Kelly S., Lyyttinen K., Rossi M., Meta Edit+: A fully configurable, multiuser and multi-tool CASE and CAME environment, Proc. CAiSE'96 Conf., Springer Verlag, 1996 Leh87 M. M. Lehman, Process models, process programming, Programming support, Proccedings of the 9th Int. Conf. on software engineering, Monterey, California, USA, 1987 Lon93 J. Lonchamp, A structured Conceptual and Terminological Framework for Software Process Engineering, Proc. Int Conf. on Software Process, 1993 Lub93 M. Lubars, C. Ports, C. Richter, A Review of the State of the Practice in Requirements Modeling, Proc. Int. Symposium on Requirements Engineering, 1993. Mer91 Merbeth G., Maestro II- das intergrierte CASE-system von Sofllab, CASE systeme and Werkzeuge (Ed. H. Balzert) BI Wissenschaftsverlag, pp 319-336,1991
23 Mic83 R. S Michalski, A Theory and Methodology of Inductive Learning, Atificial Intelligence, Vo120, No 2, 1983 Oiv92M. Oivo, V. R. Basili, representing software engineering model : the TAME goal oriented approach, IEEE Transactions on Software Engineering, Vol. 18, N ~ 10, 1992. OU88 T. W. Olle, J. Hagelstein, I. MacDonald, C. Rolland, F. Van Assche, A. A. Verrijn-Stuart, Information Systems Methodologies : A Framework for Understanding, Addison Wesley (Pub.), 1988. Ost87 L. Osterweil, Software processes are software too; Proc. 9th Int. Conf. on Software Engineering, IEEE Computer Societ, Washington, DC, 1987, pp2-13 Poh92 K. Pohl, Quality information systems: Repository for eveloving process
models, Aachen Informatik, Beichte 92-3 7-RWTH, Aachen. Poh93 K. Pohl, The three dimensions of Requirements engineering. In Proc. of the 5th Int. Conf. on advanced Information Systems Engineering, pp. 275-292, Paris, France, June 1993. Springer-Vedag. Pli95 V. Plihon, C. Rolland, Modelling Ways-of-Working, Proc 7th Int. Conf. on Advanced Information Systems Engineering (CAISE), Springer Verlag, 1995. Pot89 C. Potts, A Generic Model for Representing Design Methods, Proc. 1 lth Int. Conf. on Software Engineering, 1989. Pra96 : N. Prat, Using Machine learning techniques to Improve Information Systems Development Methods, 2nd AIS Americas Conf. on Information Systems, Phoenix, USA, 1996. Pra97 N. Prakash, Towards a formal definition of methods, in Requirements Engineering, Vol. 2, N ~ 1, 1997. Pre95 Pree W., Design Patterns for Object-Oriented Software Development, Addison Wesley, 1995 Pri87 R. Prieto-Diaz, P. Freeman, Classifying Software for reusability, IEEE Software, Vol. 4, N ~ 1, January 1987. Ram92 B. Ramesh, V. Dhar, Supporting Systems Development by Capturing Deliberations During Requirements Engineering, IEEE Trans. on Software Engineering, Vol 18, No6, 1992. Rie89. C. riesbeck, R. Schank, Inside Case-based Reasoning, Erlbaum(ed.), Northvale, New Jersey, USA, 1989 Ro193 C. Rolland, Modeling the Requirements Engineering Process, Information Modelling and Knowledge Bases, IOS Press, 1993. Ro194a : Rolland C., A Contextual Approach to modeling the Requirements Engineering Process, SEKE'94, 6th Int. Conf. on Software Engineering and Knowledge Engineering, Vilnius, Lithuania, 1994 Ro194a RoUand C., Grosz G., A General Framework for Describing the Requirements Engineering Process, C. IEEE Conf. on Systems Man and Cybernetics, CSMC94, San Antonio, Texas, 1994 Ro194b C. Rolland, Modelling the evolution of artifacts, In Proc. of the first Int. Conf. on Requirements Engineering, April, 1994. Ro195 C. Rolland, M. Moreno, C. Souveyet, An approach for beginning ways of working, In Information System Journal, Vol. 20, N ~ 4, 1995.
24 Ro196a Rolland C., Plihon V., Using generic chunks to generate process models fragments in Proc.of 2nd IEEE Int. Conf. on Requirements Engineering, ICRE'96, Colorado Spring, 1996 Ro196b C. Rolland, N. Prakash, A proposal for context-specific method engineering, IFIP WG 8.1 Conf. on Method Engineering, Chapman and Hall, pp 191208, 1996. Ro197 C. Rolland, A Primer For Method Engineering, In Actes du congr~s Inforsid 97, Toulouse, France, June 1997. Ro198 C. Rolland, C. Ben Achour, C. Cauvet, J. Ralyt~, A. Sutcliffe, N.A.M. Maiden, M. Jarke, P. Haumer, K. Pohl, Dubois, P. Heymans, A proposal for a scenario classification framework. To appear in Requirements Engineering Journal 3:1, 1998. Ros91 T. Rose, M. Jarke, M. Gocek, C. Maltzahn, H. W. Nissen, A Decision-based Configuration Process Environment, IEEE Software Engineering Journal, 6, 3, 1991 Roy70 Royce W.. W. : Managing the Development of Large Software Systems; Proc. IEEE WESCON 08/1970 ISis96 S. Si-Said, C. Rolland, G. Grosz, MENTOR :A Computer Aided Requirements Engineering Environment, in Proc. of CAiSE' 96, Crete, GREECE, May 1996. Sis97 S. Si Said, Guidance for requirements engineering processes. Proc. of the 8th Int. Conf. and Workshop on Database and Experts System Application DEXA'97, Toulouse, 1-5 September 1997. Tom94 K. Tominaga, T. Tokuda, Constraint-Centered Descriptions for Automated Tool Invocation, IEEE Asia-Pacific Software Engineering Conf. (APSEC), 1994, pp92-101. Van96 K. Van Slooten, B. Hodes, Characterising IS development project, IFIP WG 8.1 Conf. on Method Engineering, Chapman and Hall, pp 29-44, 1996. Wi183 Wilenski, Planning and Understanding, Addison Wesley (Pub.), 1983. Wir90 Wirfs-Brock J., Johnson R., Surveying current research in Object-Oriented Design, Communications ofACM, Vot. 33, No9, 1990 Fei93P. H. Feiler, W. S. Humphrey, Software Process Development and Enactment: Concepts and Definitions, Proc. 2nd Int. Conf. on "Software Process", 1993.
Aligning Legacy Information Systems to Business Processes Panos Kardasis & Peri Loucopoulos Department of Computation Department U.M.I.S.T. P.O. Box 88, M60 1QD Manchester, UK {kardasis I pl} @co.umist.ac.uk Abstract Recent years have experienced a growth in demand for re-engineering legacy information systems. The complexity of a development endeavour leading to migration of a legacy system stresses the need for a systematic supporting approach. We argue in this paper that such an approach should facilitate (a) documentation of dependencies between business processes and supporting IS in a way that changes in the business level are reflected on system specifications and (b) quantification of effects of business changes on the associated migration from legacy systems so that alternative solutions can be systematically evaluated and the optimal solution can be chosen. In this paper we put forward an approach to meeting these two objectives based on the confluence of two technologies: Enterprise Knowledge Modelling and Knowledge Discovery in Data. The approach is demonstrated using examples from a project involving a banking application. 1
Introduction
Over the past decade, continuous challenges have been made to traditional business practices. Many institutions, companies and virtually all industries have been forced into reactive patterns of change in order to remain competitive. This effort has witnessed the disabling effects that the build-up of legacy systems has on such change. Immense size and criticality in the overall business operation, use of inappropriate and obsolete hardware, poor database services, lack of interface among applications and presence of unstructured and undocumented patches are only some of the typical characteristics of legacy systems Brodie and Stonebraker 1996. As an effect, migration from legacy environments is certainly not a trivial process, while it may become extremely expensive and time consuming. Projects aiming at the replacement of legacy Information Systems (IS) by totally new ones tend to fail for a number of reasons: 9
Specifications for the legacy systems rarely exist, relevant documentation is outof-date or lost, and thus, the only source of information is the system code.
9
Too much effort is spent on the analysis phase, which may never end, either because of the complexity of the system, or because of the ineffectiveness of the analysis approach.
9
The replacement of the IS raises the need for business changes, which are rarely welcome. The fear of change in working practices, techniques and enabling
26
technologies contribute to enormous resistance. The large migration projects become even larger as everybody in the company is eventually found to be either involved or affected. This makes the situation uncontrollable and the project vulnerable to termination. The above observations stress the need for a systematic approach to assist system migration. Such an approach should support: 9
Understanding of the enterprise in terms of its operations and resources, in order to provide a solid background for analysing the system, and for assisting the coordination of business changes dictated by the migration project.
9
Documentation of dependencies between business processes and supporting IS in a way that changes in the business level are reflected on system specifications, and vice versa.
9
Quantification of effects of business changes on the associated migration from legacy systems so that system migration strategies can be systematically evaluated and the optimal solution can be chosen.
In this paper we put forward an approach to meeting these objectives based on the confluence of two technologies: Enterprise Knowledge Modelling and Knowledge Discovery in Data. The term 'enterprise knowledge modelling' refers to the set of techniques for describing the structure and business processes of an enterprise, its missions and objectives together with the way that these objectives may be operationalised onto system components Bubenko, Rolland, et al 1994; Loucopoulos 1994; Loucopoulos and Katsouli 1992; Loucopoulos and Kavakli 1995; Rolland and Grosz 1994. The term ' knowledge discovery in data' refers to correlations between data variables, identification of rules, and classifications implicitly contained in large amounts of corporate data Matheus, Chan, et al 1993; Yoon and Kerschberg 1993. Enterprise knowledge modelling provides the basis for developing models of current business processes and objectives for change, including changes to the business goals and business rules. Knowledge Discovery in Data is used for investigating the behaviour of the legacy IS in terms of its operational data and the way that such data is presently being used within the chosen business processes; it can also be used for identifying behavioural patterns which may give rise to new business processes Dhar and Tuzhilin 1993. The approach advocated in this paper is underpinned by three key activities: . Modelling of enterprise objectives, rules, and processes - for describing both the AS-IS and the TO-BE situations. . Analysis of legacy IS - for discovering the actual behaviour of the system against a set of defined business criteria.
27 3. Matching business knowledge models to results from analysis of legacy IS - for identifying the necessary changes to the IS (primarily) but also potential changes to the business processes themselves. These three activities represent the backbone of the paper. The discussion is also based on practical grounds by considering an industrial application within the banking domain. Following a short introduction of the business application and of the background modelling approach (section 2), the paper discusses the modelling activity (section 3), the knowledge discovery activity (section 4) and the integration activity (section 5), using examples from the banking application in order to demonstrate the issues being discussed. Finally, the paper concludes with a number of observations (section 6).
2
The Banking Application
The examples presented in this paper are part of a large project aiming at enabling customer profiling in the banking sector. The part of the application in which this paper is confined is one particular business area namely the marketing of loans, hire purchase agreements, and preference accounts through a dense network of local sales representatives and external agents. The critical business functions deal with: 9 Targeting of the customer base and contacting customer target groups through massive mailings 9 Contacting individual customers in order to make offers and agreements 9 Evaluating customer applications by considering both the profitability and the risk of the proposed agreements These main functions are supported currently by a legacy IS, an abstract view of which is shown in the diagram of Figure 1. The first process is supported by the 'TCS' system (Figure 1) which alerts sales representatives about customers that need to be called each time. Offers to existing customers are generally aligned to previous interaction between the customer and the bank, the history of which is provided accordingly by the system. However, the decision about the type of product to be offered and the agreement terms are left to the sales representative himself. The second process is supported by the component shown in Figure 1 as 'Application Systems'. This component is composed of many application programs that collectively deal with the scoring of customer requests according to customers' personal details, behaviour from previous transactions with the bank and references from external agencies. Finally, the 'Marketing Database' brings together the contents of all other database systems in order to facilitate decision making for massive mailing campaigns. The majority of the systems supporting the current situation were developed and put to work within the past three decades or were occasionally appended to the existing systems in an undocumented, ad-hoc manner.
28
Figure 1: The current situation The legacy infrastructure of the organisation consists of several databases, interfaces for inserting customer details and performing transactions of various types and applications for processing data and for accessing external sources of information. The heterogeneity of these legacy systems is a source of major problems including tremendous complexity, lack of sufficient documentation, co-existence of different functions serving the same purpose within the same environment, duplication of data and also confusion about which of these data are correct. Several attempts for analysing the business in the past had revealed that the existing infrastructure constrained the effectiveness of business processes. However, drastic measures could not be taken. This is a well-known problem also met by many other large enterprises handling voluminous data records and bearing strong dependencies with their information systems [Brodie and Stonebraker 1996]. Typically, the full replacement of the legacy IS would probably raise the need for parallel changes in business processes; and in addition the data conversion for supporting the new processes might be infeasible within the time limits that the business can support being without its mission-critical IS. The management of the particular organisation opted for the integration of enterprise knowledge modelling and knowledge discovery techniques. The main business objectives with respect to any business process improvement was to ensure retention of existing customers, efficiency in approaching new customers, maximum profitability of agreements and minimum contract risk through better customer understanding. The modelling approach that was used in this business application was based on the Enterprise Knowledge Development (EKD) framework. The EKD approach brings together business process modelling, and business goals within a rationale framework.
29 Recent developments of EKD are reported in Loucopoulos, Kavakli, et al 1997; Kavakli and Loucopoulos1998. The EKD conceptual models are organised according to four viewpoints: 9
The enterprise goal sub-model expresses the concepts involved in describing
enterprise objectives, enterprise issues, and their causal relationships. 9
The enterprise actor role sub-model expresses the concepts involved in modelling business processes in terms of the actors, their roles, interactions and activities in order for a business process to meet its objectives.
9
The enterprise object sub-model expresses the concepts involved in modelling physical and informational objects for the functioning of business processes and activities of individual roles.
9
The enterprise rules sub-model expresses the concepts involved in modelling the
policy, and constraints, laid down by the enterprise in the way that processes, roles and resources may behave in the enterprise.
3
EnterpriseModelling for the Banking Application
It is often argued that enterprise knowledge modelling constitutes a 'top-down' conceptual modelling activity whereby domain experts articulate their views about existing and future business situations and requirements. This was indeed the case for the banking application, as banking personnel participated in the process of defining and validating the models for the main business functions introduced in section 2. In addition, there was a significant amount of 'bottom up' modelling since much of the deliberation constituted the analysis of system functionalities. The study of the bank's support systems facilitated the understanding of business functions, regarding their purpose, and the way employees interact with each other, with customers and with the systems themselves. Much information was derived from examining existing IDEF0 documentation of system processes, interface screens and database contents. Nevertheless, it is our premise that this 'bottom-up' view very seldom is accurate since the actual functionality of the system is rarely that which is documented in specifications derived perhaps many years earlier. On the contrary, the knowledge discovery activity, which looks at the behaviour of the systems data, provides a safer understanding of the systems functionality (more about this in section 4). The enterprise knowledge modelling activity for the banking application considered a number of interrelated components, each corresponding to the EKD sub-models. These were: What is the purpose of each process? Who are the actors and what roles do they play within each process? What are the business objects required by the business system? What are the business rules that dictate the behaviour of the business processes?
30
4.1
Defining business goals
Regarding the first question, the business goals that guide the functioning of the company were defined as shown in Table 1. Table 1
The legacy information systems (shown in Figure 1) are support systems for the corresponding business processes presented in the previous table. A set of goals for change summarised in the goal graph of Figure 2 reflects the need for improvements both in business processes and their support systems.
Figure 2: The bank's goals for change
4.2
Defining business roles
Given that the overall aim of the project was to achieve customer profiling for understanding customers better, and for customising products and services to their needs and potentials, we concentrated on processes a, b and c of Table 1 for which we identified the involved actors (Table 2):
31 Table 2
Details about the roles that the aforementioned actors play in order to achieve the bank's business goals were derived from the study of the IDEF0 models. The elaboration of all the available information resulted in a number of Role-Activity diagrams [Ould 1995]. The portion presented here, reference the roles of process "Underwriting Customer Applications" which are depicted in a RAD-like notation:
Figure 3: The role-activity diagrams
32 4.3 Defining business objects The bank's business objects were mainly derived from various database tables summarised in Table 3. Table 3
After identifying the principal business objects and their interrelations, we generated the business object model (Figure 4) referring to current enterprise informational entities. The latter have been associated logically, although they are derived from flat database tables, where all the attributes were interrelated through a common database key. Applicant's Contact
Figure 4: The main components of the business object model
4.4
Defining business rules
The dynamic aspect of business processes was demonstrated in terms of role interaction business rules, which are statements that control the co-ordination of activities within business roles. Activity co-ordination rules were represented as "WHEN... IF... THEN..." statements. The "WHEN..." part contains triggering events and controls, also referenced by one or more role-activity diagrams. The "IF..." part contains the preconditions for the invocation of an activity. Preconditions are logical expressions between object controls, i.e. comparison of object attributes to certain values. Finally, "THEN..." contains the activities to be initiated when the triggering event occurs and if all the preconditions are fulfilled.
33 All the identified role-interaction rules for the specific application fall into the following three categories: Non-persistent guidelines of strategic planning and marketing decisions. The term "non-persistent" implies that decisions regarding the customer groups to be contacted and the types of products to be offered are regularly revised according to market tendencies and customers' demand. Business rules for determining which individual customers need to be contacted and when. These rules are built into the telemarketing support system. The process of contacting customers by phone is also supported by rules regarding the types of products and services that can be offered each time. Other business rules deal with the specification of agreement terms (interest rates, duration of repayment, etc.). Given that sales representatives offer services on behalf of other financial corporations, they are also bound by different rules dealing with the cross-selling of products. 'Decline' and 'pend' rules applicable in underwriting customer applications. When customers do not fulfil the conditions of the 'decline' rule set, their applications are rejected immediately. When, the conditions of the 'pend' rule set are satisfied, the application is forwarded to the underwriter for manual evaluation. The underwriter also considers a number of implicit rules, in order to coestimate the credibility of the customer, and his profitability for the bank. Examples of modelling business rules according to the EKD textual and diagrammatic notations follow:
4
Knowledge Discovery for the Banking Application
Knowledge Discovery in Data (KDD) has been defined as the non-trivial process of identifying valid, novel, potentially useful and ultimately understandable patterns in data. The KDD process usually involves data preparation, knowledge evaluation and refinement. The step of applying algorithms for extracting patterns from data is referred to as data mining. The existence of a data warehouse is a necessary "preprocessing" condition for developing knowledge discovery applications. Knowledge discovery usually starts with understanding the outcome variables, as well as the overall application universe (i.e. the data populated in the warehouse) [Tej Anand 1995]. In the case examined here, both steps were facilitated by the business object model presented in the previous section, which groups the available data attributes in an unambiguous and flexible way.
34 The mining exercise [Keane, Murray, et al 1997; Filippidou, Keane, et al 1998] was conducted upon a sample of the overall bank's data for loans and hire purchases. As it can be seen in Figure 4, the examined data constitute history of customer applications. Apart from current contact details (phone number and address) and history of contacts (dates of calls and mailings), all the other information describes the profile of the customer the time that he submitted his application for a certain product. Applications are related to customers':
Other critical attributes like B e h a v i o u r S c o r e for the customer a n d Reason For D e c l i n e for the application are directly assigned to A p p l i c a t i o n . The c u s t o m e r is only related to his persistent personal details, be they his/her name and surname, sex and date of birth. The outcome of data mining was a set of classification decision trees [ATTAR 1996], which represented the relation between one or more input data variables and a predefined output variable. The classification output variable was the Reason F o r D e c l i n e , reflecting whether an application has been accepted, rejected or just has not resulted in agreement, and the reasons why. The information represented by the generated classification decision trees is equivalent to statements of the following structure:
From the generated results, influences of different attributes were determined and were used to inform the enterprise knowledge models and potentially to modify the systems and the business way of working.
Integration Between Enterprise Knowledge Models and Discovered Knowledge The outcome of the activities described in sections 3 and 4 is: (a)
A set of specifications for the bank's business processes.
(b)
Informal descriptions and models presenting the purpose, structure, functionality and constraints of their support IS.
35 (c)
A number of statements reflecting the conclusions from the data mining exercise (customers classification and clustering of the customer base).
We claim that none of these views is sufficient for driving the transformation of the enterprise and/or the migration of their legacy IS. However, the integration of the gained knowledge (both developed and discovered) provides an overall picture of how the bank's business goals can be operationalised. Table 4 relates several suggestions for the improvement of the bank's legacy IS with the business goals that these improvements may fulfil. Table 4
Figure 5 provides a high-level view of the bank's business processes also considering the rule sets that are used in each case. Currently, the 'decline' and 'pend' rule sets are the only explicitly expressed and documented sets of business rules within the bank's functions. The 'marketing strategy' represents approaches and tactics in organising massive mailings, being based on the analysis of the bank's marketing database contents. The 'algorithm of the telemarketing support system' refers to several behaviour characteristics of the " r c s ' system. Finally, the 'scoring policy' deals with the current method for estimating
36 customers' credit and behaviour scores. This high-level view, depicted graphically in Figure 5 is the current way of working. To demonstrate the impact of the discovered knowledge on re-designing and improving the current way of working there were three topics of relevance: (1) Correlation between input and output variables; (2) Validation of existing business and system functions; and (3) Management of 'rule' related knowledge.
Figure 5: High-level view of the bank's processes
Correlation between input and output variables The discovered knowledge resulted in statements of the following type: of customers
"Group x
is r e l a t e d to an a p p l i c a t i o n a c c e p t a n c e p r o b a b i l i t y
z~ . Decision-makers are potentially aware of the existence of such rules, due to their intuition and experience. However, it is very useful to move from fuzzy estimations to more explicit and quantified correlations like the ones described above. These new rule expressions can be applied in targeting customers more accurately, by aligning the number of mailings that the company can afford to send with the usual response rate of customers. Other rule statements of the type " G r o u p x o f c u s t o m e r s is r e l a t e d to an a p p l i c a t i o n a c c e p t a n c e p r o b a b i l i t y z% f o r p r o d u c t y" may contribute to the maximisation of customer response by
aligning offers with customer needs. These new rules are grouped under the 'customer targeting rule set' (Figure 6), while their integration in the new IS will satisfy goals 1, 5, 8 and 9 of Table 4. Similar clusterings may facilitate understanding of individual customers, so that maximum response to offers is ensured. A number of data mining extracts may result in business rules ('contacts co-ordination rules') for determining which is the best time for contacting a certain type of customer, and which are the most suitable products to be offered. In this way intelligence is added to the telemarketing support system being responsible for the co-ordination of telephone contacts (goals 1 and 8 in Table 4).
37
Validation of existing business and system functions The knowledge discovery experiments showed that several operations (on business and system level) are not that much dependent on certain data attributes as previously thought. This observation raised the need for revisiting the corresponding processes and legacy support systems, and for improving them in case the problem is due to inaccurate input data, or erroneous algorithms for processing them, rather than to inaccurate data mining extracts. Such improvement efforts constitute an indicative example of cases where knowledge discovery practices are used for validating the current business and systems. Figure 6 reflects changes of the bank's processes advocated by the previous observations:
Figure 6: High-level view of the future situation Management of rule-related knowledge Business policies, regulations, guidelines, and also discovered knowledge expressions within a large enterprise can be considered as a complex net of interactions, given that they all make reference to the same business objects and activities. Appropriate tools can facilitate the management of this knowledge, by identifying the dependencies and conflicts between rule expressions and by propagating potential changes (according to newly discovered knowledge or new business strategic decisions) along the affected business processes, the people that need to be informed and involved and the systems that need to be modified or customised accordingly (goal 2 in Table 4). 6
Conclusions
Migration of legacy IS is becoming increasingly important as organisations face significant pressures for change. Such a migration however, is a complex undertaking, requiring the commitment of large resources with uncertain results at the outset. Therefore, understanding the relationship between the needs of the business domain and the capabilities of their support IS is of critical importance. Such an
38 understanding would lead to better planning with a more evaluative way of assessing the risk of the migration task. In this paper we have put forward an approach to developing this understanding through the alignment of the business itself to its legacy IS. Central to this approach is the confluence of two techniques namely, enterprise knowledge modelling and knowledge discovery from data. The former is concerned with the development of models pertaining to business objectives, business processes and business rules whereas the latter is concerned with the development of models from the discovery of business behaviours from the existing IS. We have presented a way of utilising these techniques and demonstrated their utility in terms of a large banking application. By integrating the two sets of models it is possible to: (a) identify those parts of the legacy IS that require improvement to the extent that they will meet the stated objectives for change and (b) improve the business knowledge in terms of opportunities that may be available through the innovative exploitation of hidden knowledge.
7
Acknowledgements
The work reported in this paper has been partly supported by the commission of the European Union under the ESPRIT programme. The authors wish to acknowledge the assistance of Mr John Keane, Mrs Sofia Svinterikou and Mr Bob Scott in the insights of the data mining results.
8
References
ATTAR Software Limited. (1996) XpertRule Profiler: Knowledge from Data, 1996. Brodie, M. and Stonebraker, M. (1996) Migrating Legacy Systems, Morgan Kaufmann Publishers Inc, San Francisco California, 1996. Bubenko, J., Rolland, C., Loucopoulos, P. and de Antonellis, V. (1994) Facilitating "Fuzzy to Formal" Requirements Modelling, IEEE International Conference on Requirements Engineering, 1994. Dhar, V. and Tuzhilin, A. (1993) Abstract-Driven Pattern Discovery in Databasez, IEEE Transactions on Knowledge and Data Engineering, Vol. 5, No. 6, December, 1993, pp. 926-938. Filippidou, D., Keane, J., Svinterikou, S., Murray, J., (1998) Data Mining for Business Process Improvement: Using the Hyperbank Approach, PADD98, 26-28 March 1998, London, U.K. Kavakli, V. and Loucopoulos, P. (1998) Goal Driven Business Analysis: An Application in Electricity Deregulation, CAiSE*98, 8-12 June 1998, Pisa, Italy. Keane, J., Murray, J., Scott, B. and Svinterikou, S. (1997) Preliminary Analysis of Data Mining Results, UMIST, Hyperbank WP3/T3.5/U/01, 1997. Loucopoulos, P. (1994) The F3 (From Fuzzy to Formal) View on Requirements Engineering, Ing6nierie des Syst~mes d'lnformation, Vol. 2, No. 6, 1994, pp. 639-655. Loucopoulos, P. and Katsouli, E. (1992) Modelling Business Rules in an Office Environment, ACM SIGOIS, No. August, 1992.
39
Loucopoulos, P. and Kavaldi, E. (1995) Enterprise Modelling and the Teleological Approach to Requirements Engineering, International Journal of Intelligent and Cooperative Information Systems, Vol. 4, No. 1, 1995, pp. 45-79. Loucopoulos, P., Kavakli, V., Prekas, N., Rolland, C., Grosz, G. and Nurcan, S. (1997) Using the EKD Approach - The Modelling Component, UMIST, The ELEKTRA Project WP2/T2.1/UMIST/1, April 1997, 1997. Ould, M.A., (1995) Business Processes: Modelling and Analysis for Re-engineering and Improvement, John Wiley & Sons Ltd, U.K., 1995. Matheus, C.J., Chart, P.K. and Piatetsky-Shapiro, G. (1993) Systems for Knowledge Discovery in Databases, IEEE Transactions on Knowledge and Data Engineering, Vol. 5, No. 6, December, 1993, pp. 903-913. Rolland, C. and Grosz, G. (1994) A General Framework for Describing the Requirements Engineering Process, IEEE International Conference on Men, Systems, and Cybernetics, IEEE Computer Society Press, San Antonio, Texas, 1994. Tej Anand, A.T.G.I.S. (1995) Commercial Knowledge Discovery Applications, KDD95, Montreal, Canada, 1995. Yoon, J.P. and Kersehberg, L. (1993) A Framework for Knowledge Discovery and Evolution in Databases, IEEE Transactions on Knowledge and Data Engineering, Vol. 5, No. 6, December, 1993, pp. 973-979.
Automated Reverse Engineering of Legacy 4GL Information System Applications Using the ITOC Workbench John V. Harrison and Wie Ming Lim Centre for Software Maintenance Department of Computer Science and Electrical Engineering The University of Queensland, Brisbane, QLD 4072, Australia E-mail: {harrisonlwieming} @csee.uq.edu.au
Abstract. Most contemporary fourth-generation languages (4GLs) are tightly coupled with the relational database and other subsystems provided by the vendor. As a result, organisations wishing to change database vendors are typically forced to rewrite their applications using the new vendor's 4GL. The anticipated cost of this redevelopment can deter an organisation from changing vendors, hence denying it the benefits that would otherwise result, for example, the exploitation of more sophisticated database technology. If tools existed that could reduce the rewriting effort, the option of changing database vendors would become more economically feasible. The ITOC project is a large collaborative research initiative between the Centre for Software Maintenance at the University of Queensland and Oracle Corporation. The primary goal of the project is to develop tools to assist in the migration of 4GL information system applications. A tool resulting from the project has been utilised to recover design information from several deployed commercial applications. This paper describes the tool, evaluates its performance when applied to these applications and provides insight into the development of "industrial strength" re-engineering tools.
1. Introduction There has been a significant investment in the development of information systems built using fourth-generation programming languages (4GLs). Although there have been considerable advances in the design of both 4GL languages and their associated development environments, most are still proprietary. There are no open industry standards, as there are with third-generation programming languages (3GLs) such as COBOL. The dependency of the customer on a particular vendor often prevents the customer from realising the benefits of newer technologies without incurring a very large redevelopment cost. Thus, in order to make application migration economically feasible, it is important to develop tools which will assist in the migration process. Most work in information system reengineering addresses either the data repository alone, or different aspects of migrating 3GL applications that access a network, hierarchical or some other legacy data repository. While there remains significant demand for tools that assist in this domain, the pace of technological advance in database systems, as well as the high level of competition amongst
42 database vendors, has resulted in 4GL-based information system applications now being considered as "legacy" by their organisations. The ITOC Design Recovery Tool (Harrison, et al., 1995, Berglas and Harrison, 1997, Harrison and Berglas, 1997, Harrison, et al., 1997) was developed collaboratively by the Centre for Software Maintenance at The University of Queensland, and Oracle Corporation, and has been deployed to assist with the recovery of both the application semantics and the static schema definition from Ingres ABF rM 4GL applications. The recovered design components are loaded into the Oracle Designer 2000 TM CASE repository, and can then be used to forward engineer an application in several representations, for example, HTML, Visual Basic and Oracle Developer 2000 TM (a "form-based" implementation environment). To the best of our knowledge, this is the first 4GL design recovery technology that has been implemented beyond a prototype stage. Existing techniques that we believed would address part of the design recovery task were determined to be insufficient when applied to deployed commercial applications. Consequently, we concur with the observations and conclusions made by (Blaha and Premerlani, 1995), which state that many techniques fail to be effective when confronted with the idiosyncrasies that occur in real applications. The next section provides a general introduction to the characteristics of 4GL information system applications using as examples the source and target development environments selected for the project. Following that, we describe the basic structure and functionality of the too1. Section four presents our results, experience, and our evaluation of the applying the ITOC too1 to legacy applications. We then describe related work and conclude with a summary and future research direction.
2. Information System Application Characterics 4GL-based information systems are comprised of a user interface, application logic and a relational database management system (RDBMS). Much of the power of 4GLs is derived from the recognition that most user interfaces can be modelled as forms, which can be viewed as electronic representations of their paper-based counterparts. Forms typically contain a number of fields that correspond directly to columns in the database. The 4GL development enviroment environment is utilised to provide the (considerable) amount of user interface functionality necessary to faciliate cursor movement, e.g., to monitor cursor movement from field to field and perform basic validation. Fragments of procedural 4GL code are invoked when complex validation is necessary. For example, consider the sample Ingres ABF "order entry" form illustrated in Figure 1. In order to implement this form the programmer would declaratively specify the name, data type and position of each field, and also the fixed "boiler plate" text such as the string "Invoice Num". Most 4GLs will then automatically implement logic that permits the user to move the cursor from field to field, ensure that alphabetic characters are not entered into numeric fields, and enable multiple records such as the products purchased on the order to be scrolled in the scrolling region. Automating this functionality is an advantage of using 4GLs because implementation using a 3GL is both tedious and error prone. However, 4GL code is then required to
43
perform higher level validation and processing such as ensuring that both the Billing and Shipping Customer Numbers are valid, retrieving the Customers' Name, and calculating the "Total Delivered Cost". mI
M~Wl. ~c~
J
I n v o z c e Num: ~ I s s u e D a t e : Wednesdau. 2 0 t h December Remark: l e h v e r y Notes: D e l i v e . t o the back door. knock ~ x c ~ and ask Cot .
.
.
.
.
. . . Shipping Num: ~ Name: UnzveesitU o~ Queensland Address: St L u c i a , B~isbane, QLD, A u s t r a l i a Post Code: 4072
Next(l)
Status:
OUOTEB
.
- S h l p p l n g Customer.
Product Mum Product Name
1995
r--B1111ngC u s t o m e ~ - l B i l l i n g Num: Name: Oracle CASE Group Parent: Oracle Corporatzon
|
Total Delivered Cost:
Ordered Qtg MznlmumQtg!gel~vered ~tg P r i c e
Delete Row(g) Delete Master(3)
Update Mastee(4)
$ 74481.00 Del T o t a l
End(PF3)
Figure h A Sample Ingres ABF Form The Ingres 4GL development environment requires a considerable amount of 4GL code to be written to retrieve records from the database and display them on the form, and then to update changed information. An Ingres ABF application is comprised of ABF frames. Each frame is comprised of user interface items that appear on a particular screen, termed an ABFform 1, and code which implements the application logic associated with that particular screen. The code contains embedded SQL, which is typically used to retrieve values from the database into fields on the screen, and vice versa. While 4GLs reduce the amount of code required to build user interfaces, CASE tools can further improve productivity by also automating the code that is required to move data between the form and the underlying database. A developer using a CASE tool can avoid the necessity of writing code explicity to implement certain application logic and creating the user interface of each module. These components are generated automatically from a very high level description of the application, which is input into the CASE repository. For example, the Oracle CASE tool allows forms to be generated automatically based on a "Module Data Diagram", which is made up of "Module Detail Table Usages" (MDTUs). A MDTU shows how a database table is to be used to construct a specific module. Within a MDTU, "Module Detail Column Usages" (MDCUs) show the usage of individual database columns within a particular module. Figure 2 shows a Module Data Diagram corresponding to the application illustrated in Figure 1. Each rectangle on the diagram represents a usage of a table by this module, namely a MDTU. The layout of the MDTUs in the Module Data Diagram is important as it represents the relationship between the underlying database tables. The relative placement of the MDTUs in a module ultimately determines the appearance and functionality of the generated target application. 1Notethat the term "form"is overloadedhere. IngresABF "frames"correspondto the formconstructdescribed earlier. Thisis a simpleexampleof the lack of standards in this domain.
Figure 2: Module Data Diagram for the Invoices Frame The CASE environment supports two types of relationships, or links, between MDTUs. These relationships are termed master-detail and look-up, which are terms that are commonly used by practitioners using different 4GL environments. Both master-detail and look-up relationships are only specified if a foreign key exists between the database tables on which the MDTUs are based. For example, a "masterdetail" relationship exists between the INVOICES and ITEMS MDTUs in Figure 2 (note the placement of the former MDTU being above the latter). This relationship implies that a foreign key exists between the ITEMS and INVOICES database tables. Similarly, a "look-up" relationship, such as the one between the CUSTOMERS and INVOICE MDTUs in Figure 2, can only be defined if a foreign key exists between the INVOICES and CUSTOMERS database tables. Look-ups are positioned to the right of the referenced MDTU on the diagram. On the basis of the above Module Data Diagram (MDD), the CASE tool can generate an application in several representations, such as the Oracle Developer 2000 TM, HTML and Visual Basic. These representations would offer equivalent functionally to the Ingres implementation illustrated in Figure 1. The MDD graphical description is all that need be specified to enable the CASE tool to automatically generate the code required to select Invoice and Items rows from the database, look up and validate their Customer and Product Numbers, and update the database in response to user input.
3. Design Recovery From Information Systems The ITOC design recovery process involves the extraction of information from an Ingres relational database application, and the conversion of this information into Oracle CASE repository elements. An overview of the process appears in Figure 3. The ITOC tool uses information from the Ingres relational database schema as obtained from the database catalogue, the user interface (screen) definitions as extracted using the Ingres "copyapp" utility and 4GL procedural code that is contained within the source, i.e., ".osq", files. After processing, the tool loads the
45 recovered design information into the CASE repository. Executable applications, in various representations, can then be created automatically using code generators. The processing steps implemented by the ITOC tool are shown in detail in Figure 4. Each step produces objects that are defined by the ITOC analysis schema, and then used as inputs to the next step. The output of the tool are CASE elements that define the schema and modules needed to construct the Module Data Diagrams, and other design information, that corresponds to the source application. The following sections provide an overview of the processing performed at each step. Many details have been omitted due to space limitations. A more comprehensive description can be found in (Harrison and Berglas, 1997).
Figure 3: Overview of the ITOC Design Recovery Process
Figure 4: Steps of the ITOC Design Recovery Process 3.1 Parse and Link The first step is to parse the source code and link uses of each identifier to its definition. This was performed using Reasoning System's RefineryTM environment, including the RefineTM programming language and the DialectTM compiler compiler. We found that, unlike many other compiler compilers, in addition to parsing the
46
source file given a formal grammar of the source language, Dialect automated the production of an abstract syntax tree. The Abstract Syntax Tree (AST) output consists of objects which represent instances of terminal and non-terminal nodes in the grammar, together with derived information such as the links to identifier definitions. The Language eXtension WorkBench (LXWB) (Peake, 1996) was used to extend Dialect to unify the definitions of the formal grammar and the objects that form the AST. 3.2 AST Abstraction Once the AST is created, it is compressed into a high level abstraction which is more suitable for further analysis. Defining an explicit abstraction of the source code enabled different syntactic structures to be processed in a consistent manner. For example, while most queries are represented by explicit SQL statements within the 4GL code, it was discovered that a common Ingres coding practice is to write 3GL programs which perform implicit queries by scanning plain text tables of static data on the client side of a client/server environment. AST abstraction enabled these procedure calls to be abstracted, and subsequently treated exactly the same as if they were SQL queries despite the fact that they have a radically different syntactic structure. The results of this processing phase are recorded in the AST Abstraction Schema, which is used as input into the following phase.
3.3 Data Flow Analysis The ITOC design recovery process relies on analysis of data flow in the Ingres source code. The data flow analysis is heuristic-based and differs from the conventional use of data flow analysis (Callahan 1988, Marlowe and Ryder 1990, Ryder et. al. 1990). The heuristic we adopted assumes that if a data flow exists from a source object A to a target object B, then the value of object A is coupled, and hence considered equal, to object B. Based on our experience applying the tool, which is described below, we found this heuristic to be effective. The data flow analysis phase consists of two sub-phases. Initially, all the direct data flows between variables and database columns are recorded. For example, the first line of the first SQL query variant in the code fragment below contains a data flow from I. Invoice_Num to the variable :Invoice_Num in the first query statement. Note that the data flows are recorded between each reference in the code to a database column, which we term query columns. Select From Where
:I n v o i c e _ N u m INVOICES I
= I. I n v o i c e _ N u m
Select From Where
:Billing_Cust_Name = C.Cust_Name CUSTOMERS C C . O / s t _ N u m = :Billing_Cust_Num
:Billing_CustNum
= I.Billing_C~st_N~m
On the basis of the direct data flows, the transitive links that exist between two query columns, as opposed to between a query column and a variable, are computed. These transitive data flows are needed to capture the implicit enforcement of foreign
47 keys, such the one between the "I.Billing_Cust_Num" and "C.Cust_Num" query columns via the ":Billing_Cust_Num" variable appearing in the code fragment. Note that this phase only computes the transitive data flows. The derivation of foreign keys is done in the following phase.
3.4 Query Analysis and Data Mining A fundamental part of the ITOC design recovery process addresses the recovery of foreign keys using information obtained from the Ingres 4GL code. This recovery is necessary as the Ingres Data Definition Language (DDL) does not support the explicit declaration of foreign keys, which form an integral part of the construction of Module Data Diagrams. Based on the data flow, both direct and indirect, recorded in the previous phase, the ITOC tool computes references between query tables, which are references to a database table appearing in a query. A reference object is created for each data flow that terminates at, or originates from, a reference to a database column that forms part of a key of a relational database table. The data flow must be from a query column, as opposed to a variable. If every part of a key is covered by a reference object, meaning there exists a data flow from some query column to each part of the given key, then a candidate foreign key is generated. The generation of the candidate foreign keys is based on heuristics about data flows. As the heuristics are not completely accurate, invalid foreign keys will be proposed. Pruning these invalid candidate foreign keys is necessary to reduces the possibility of errors in subsequent phases. The candidate foreign keys are tested using rudimentary data mining techniques applied to the original Ingres database instance. In addition to validation of candidate foreign keys, data mining is used to derive some additional information, namely: 9 Whether columns were deemed mandatory or optional by the source designer. Although Ingres contains null and not-null DDL definitions, their semantics are inconsistent with the target database environment. As a result the Ingres DDL cannot be used to determine this information directly, 9 Specific ranges for column values. For example, an Order's status might have been restricted to "QUOTE", "ORDERED", or "PAID". 9 Whether foreign keys participate in a one-to-one cardinality relationship. 9 Whether numeric fields may be contain negative values. The results of this phase are represented in the Query Analysis Schema, which would now contains objects which define the foreign keys, and other domain constraints, that were implemented in the source application.
3.5 Form Analysis Form analysis identifies which database column is used to populate each field in an Ingres frame and which fields are populated with derived values. In the CASE tool, a field can be associated with at most one database column. Ambiguities related to uniquely associating a database column, via a MDCU, to a field arise from the fact that data flows can exist between more than one query column and a given field, which we were informed often occurs in practice. For example, if a billing customer
48 can be the same as a shipping customer, then the billing customer's details will be displayed in both the billing and shipping fields on our example Invoices frame. As a result, a data flow will have been created from both the shipping and billing customer names to the shipping customer field. The goal of this phase is to determine which database column should populate the shipping customer field. The solution to resolving the above problem is based on a strategy involving numerical weighting of the relative data flows. Using weighting heuristics, which are described in detail in (Tech 1996), each field can be uniquely associated with a database column. Note however, that not every field will necessarily be associated with a database column, for example, fields which get their value from a complex calculation perhaps involving several database columns. At present, the tool only highlights the existence of such fields, and provides links to their occurrence in the source code via a hypertext report viewable using a conventional web browser. The results of this phase are recorded in the Forms Abstraction subschema, and again used as input into the subsequent, and final, processing phase.
3.6 Module Analysis Having uniquely associated fields and database columns, the Module Data Diagrams can be created. Module Analysis maps each Ingres frame to a corresponding Module Data Diagram, and determines the relationships / usages between the MDTUs for the particular module. In addition to producing a mapping between each Ingres frame and a corresponding Module Data Diagram, the tool also provides the software re-engineer with alternate design options. Through analysis it was discovered that an Ingres application will often contain several distinct forms which all perform a semantically related task. For example, one Ingres form may allow the user to read Invoice details, another may allow the editing of the Invoice and a third may allow the creation of a new Invoice. In the target 4GL, the same functionality can be implemented using one form, which allows updates, queries and insertion of new records. By examining the data flows between forms, the tool is able to suggest an alternative, more succinct representation of the source application. This phase is described in (Harrison and Berglas, 1997). The remainder of this paper evaluates the effectiveness of the ITOC design recovery process described in the previous sections. The evaluation is based on the commercial and custom built applications that have been re-engineered using the ITOC tool.
4. Evaluation of the Tool's Performance This section describes the results of applying the ITOC tool to four Ingres ABF applications. Three are commercial applications that have been deployed. The fourth was constructed by the ITOC research group to test the tool on an application that contained constructs permissible by the language but are rarely used in practice
49 according to the practitioners who participated on the project. Each application is described briefly below. The Customer Invoicing Application (CUST-INV) is the experimental Ingres ABF application, which contains various 4GL constructs that our analysis, and discussions with practitioners, indicated were obscure and only occasionally appear in a deployed commercial application. As the tool was designed to be robust in anticipation of worldwide deployment, any permissible construct had to be addressed and tested. The Curriculum Management System (CMS) is a large Ingres application (containing 120 frames) used by the Australian Technical And Further Education Institute of Australia (TAFE) to manage their curriculum. The Curriculum Monitoring Application (CMA) is a related application (10 frames) belonging to the TAFE Institute. The Contractor Monitoring System (DME) is a contractor registering and monitoring system of the Australian Department of Mines and Energy. DME contains 41 frames which record and maintain information about contractors and their licensing details. The tool's effectiveness was measured primarily on its ability to recover a number of major design components, namely Tables, Module-Detail-Table-Usages (MDTU),
Module-Detail-Column-Usages (MDCU), Foreign-Key Usages, Master-Detail Relationships (MDR), and Look-Up Relationships (LUR). Tables refer to the database tables that are referenced in a particular Ingres form. Foreign-Key Usages represent the usage of a foreign key in a given module. The remaining components were introduced in Section 2. Table 1 presents a summary of the results of the evaluation. For each application, the number of design components found in the source Ingres ABF application, by a manual line-by-line review of the code by a domain expert, appear in the columns labelled "In Source". The quantity of each design component recovered by the ITOC tool, stored in the CASE repository and also properly displayed to the developer using the CASE environment appears in the columns labelled "Recovered". The last column of Table 1 summarises the performance of the tool across all four applications. Although these figures give a reasonable indication of the performance of the tool, other more informative criteria may exist. This topic is currently under investigation. In addition to the analysis results based on the above criteria, we describe some additional features of the tool which facilitate the overall re-engineering process. 4.1 Evaluation Results The numerical results of the application of the ITOC tool on the four applications are shown in Table 1. Analysis indicated that omitted design information was due to one of only four reasons, namely missing foreign keys, missing MDTUs, errors in distinguishing integrity constraints, and errors displaying foreign keys that were recovered.
50
CUST-INV
CMA
covered
Recovered
Percentage Recovered
83
83
219
219
100%
5
30
23
205
79
46.6%
38
38
296
296
640
640
100%
14
14
78
78
295
265
92.5%
4
3
3
13
13
48
40
88%
5
3
2
17
10
154
39
31%
Recovered
In Source
coveredI
In Source
10
10
14
14.
Foreign Keys
10
10
6
MDCUs MDTUs
43
43
15
15
MDR
4
LUR
6
Tables
CMS
DME
In Source
In Source
Re-
Re-
Referred l
Table 1: Results of the Analysis of the Four Applications 4.1.1 Missing Foreign Keys As mentioned in section 3.2, we discovered that a common Ingres programming construct involved calls to 3GL procedures to retrieve code descriptions for a given code. In general, the 3GL code will contain a look-up to what is referred to as a "code table" such as the partial one illustrated in Table 2. In this example, the primary key of the code table is the columns Code_Type and Code.
Code_Type
Code
Columns
SUBJ_CODE
Subj_Dtls_l
Subj_Abbr_Name
SUBJ_CODE
Subj_Dtls_2
Subj_Description
Table 2: Example Code Table In the Ingres frame, the call to one of these 3GL look-up procedures will contain a hardcoded reference to the Code_Type, for example: c a l l p r o c C O D E _ T A B L E _ L O O K U P ( 'S U B J _ C O D E ' , BYREF (Subj_Description)) ;
subj_code,
Here the calling frame passes a literal ('SUBJ_CODE') and a variable ( s u b j _ c o d e ) to the 3GL procedure. This results in two data flows, one to each component of the primary key of the code table. Although each component of the primary key of the code table is represented as query column appearing as an endpoint of a data flow, a reference object is not created for this data flow. Recall from section 3.3 that the source and destination of each relevant data flow must be from a query column (either directly or indirectly). As one of the data flows originates from a literal, a reference object will not be created, which results in a missing foreign key. An unrecovered foreign key subsequently results in one of the master-detail or look-up relationships being missed. With reference to the results in Table 1, this case accounts for all the missing foreign keys, and then subsequently missing look-ups, in the CMA and DME applications, and also similar missing components in the CMS application.
51
4.1.2 Missing MDTUs Analysis of the CMS application revealed that all frames constructed for querying the database were of similar structure. These frames consisted of a master-detail relationship between MDTUs based on the same underlying database table. The user enters search criteria into the master frame, and the results of the query are displayed in the detail frame. The ITOC tool incorrectly corrupts the design by combining the two MDTUs into one. The error is due to incorrect formation of query table sets, which are abstractions used to represents the consolidation of similar queries and updates appearing in the 4GL code. For example, a select, update, and delete operation on the same database table, involving the same columns are represented as one query table set. This more consise representation was found to be more maintainable and would result in a better implementation after generation, as described in section 3.6. Although the query table set principle is sound in most cases, there are instances, such as the case described above, where the tool incorrectly unifies MDTUs and hence misses foreign keys and master-detail relationships appearing in the CSM application. This problem of forming correct query table sets is described in (Berglas and Harrison, 1997) and is currently under investigation. Another characteristic of the CMS application involves the population of a detail MDTU using tuples derived from a relational union of the results of several SQL queries. The version of the target CASE tool did not support the concept of relational union directly through its graphical module creation utility. As a result, in order to represent this construct in the CASE repository, the designer can either define a view definition that includes a union operation, or write a stored database (PL/SQL) procedure. At present, the ITOC tool only recovers components that are directly representable in the CASE repository, hence this construct is not recovered, which accounts for the missing MDTUs, master-detail relationships, and foreign keys that should have been recovered from the application.
4.1.3 Referential Integrity Constraints The Oracle CASE tool supports relationships between MDTUs based on foreign key constraints between database tables but does not support the more general referential integrity constraint (Elmasri and Navathe, 1994). In the DME application, there are several frames that, after design recovery, would result in MDTUs connected using referential integrity constraints, but not foreign keys constraints. An example is a look-up to a database table that references specific tuples in the target table using only part of its key. For example, consider the following ABF-variant of an SQL Select statement: Select :Endorsee_Name = E.Endorsee_Name From ENDORSEES E Where E. L i c e n c e _ N o = : L i c e n c e _ N o Assuming that the primary key of the ENDORSEES table is the combination of
Endorsee_Name and License_No, and that the variable :License_No contains a value from the LICENCES table.
The relationship between the ENDORSEES and
52 LICENCES tables is based on a referential integrity constraint, but not a foreign key constraint. As a result of the restriction in the particular version of the target CASE environment used in our experimentation, a link between MDTUs based on these two database tables cannot directly enforced using the declarative mechanism provided. Consequently, the ITOC tool does not attempt to recover this information, which accounts for the missing links we expected to encounter after manual analysis of the DME application.
4.1.4 Foreign Keys Recovered But Not Displayed The ITOC tool utilises information from the user interface definitions to create MDTUs. One such component of the definition is the field sequence. The field sequence specifies the order in which the fields were created by the application developer. As a result of manual analysis, which was verified using practioners, we observed that the fields of an ABF screen that corresponds to a column of a look-up MDTU, which we term look-upfields, occur later in the sequence than fields in ABF screen that corresponds to a master MDTU, which we term master fields. Consequently, the ITOC tool employs a heuristic based on this observation that assists in distinguishing master and look-up MDTUs. However, in the case of the CUST-INV application, an anomaly occurs. A lookup field occurs before the master field. This results in the failure of the tool to create the MDTU constraint, and also accounts for the missing look-up relationship. Note however, that the associated foreign key is recovered. It is only its usage in this module that was not recovered.
4.2 Other Recovered Design Information In addition to the above, the ITOC tool recovers other design information from the source application. This recovery, and some informal observations, are described in this section.
4.2.1 User Interface Specification When the user interface definition is declaratively specified in a 4GL environment, as is most often the case, it can be almost completely recovered. Re-use of the recovered specification can be of value to an organisation in certain circumstances. If the re-engineered system retains a similar, or identical, appearance and behaviour as the source system, fostering user acceptance of the re-engineered system may be easier. In addition, the cost to retrain end-users to use the reengineered system may be reduced. The declarative interface specification includes the position of the fields on the source application's form, the ordering sequence of the fields, the length of the fields, and the "boilerplate" text, ie., field labels, associated with each field. Text enhancement characteristics such as bold, underline, blinking, etc., are also recovered. As this information is both explicit and declaratively represented, only rudamentary information system semantic analysis is required to allow the tool to completely recover this information.
53 The recovered specification is loaded into the CASE repository, and is used to create templates, which are used for generating the user interface of the target application. Even when the source user interface is text-based and the target user interface is graphical (GUI) it can be easily determined that the design of the user interface originated from the source application. The templates can also be used to create new applications, unrelated to either the source or target, that possess the same "look and feel" as the source, hence retaining consistency.
4.2.2 Business Rule Recovery Certain expressions appearing in the application logic or declarative user interface specification are both identified, and recovered, from the source application. These expressions, termed derivations and validations, can derive the value to be stored in a field and can restrict the value that will appear in a field. These expressions represent a subset of a larger set of logical expressions that complete the representation of the semantics of the application, and are termed "business rules" by practitioners. The ITOC tool analyses assignment statements such as "A := B + C". It also analyses SQL "Select" statements such as "Select A + B, from T where C > D and (I = J or K = L)" and validation expressions such as "Minimum_Amt > Ordered_Amf'. After analysis, the tool produces a business rule report, The report contains a hyper-text (HTML) listing of each frame's source code, displaying business rules (in italics) and references to variables that are connected to either fields or query columns (in bold), which we observed indicated the existence of a business rule. The use of the report reduces the likelihood of these rules being overlooked during the manual re-implemention phase of the re-engineering process. The practitioners who completed the development of the target application after performing design recovery using the ITOC tool found the report useful but had hoped for more extensive support for business rule recovery than provided by the tool.
4.2.3 Create, Read, Update and Delete Operations Operations for creating, reading, updating and deleting database tuples, which are collectively referred to as CRUD by practioners, are the four possible types of operations that can be defined in a CASE Module Data Diagram. This information is recovered from the source application by analysing SQL statements in the application, and is stored in the CASE repository. This allows the CASE generators to produce target applications which support the same table operation modes as the source application. This information is both explicit and declaratively represented in the 4GL code, hence only rudamentary information system semantic analysis is required for full recovery.
4.2.4 Metric Report The ITOC tool contains a utility for estimating the effort required to complete reengineering of an application after design recovery using the ITOC tool is complete. The report is based on heuristics provided by the practioners that reason with the
54 number of activations appearing in a frame. Activations, which are also referred to as triggers, are event-activated procedural code segments appearing in a 4GL application. The heuristics also reason with the number of activations associated with each field and the number of occurances of a program variable in non-trivial expression. The practioners indicated that the report proved useful. A more detailed description of this report appears in (Tech 1996).
5. Related W o r k This section provides a summary of research that relates closely to the work described here. It is loosely classified into that which only attempts to analyse data, and that which also attempts to migrate application code. (Petit et al., 1994) proposed a design recovery method to extract an entityrelationship (EER) schema from a relational database extension and SQL queries. Like ITOC, the method does not require that the same attribute in different relations have the same name nor that the relations are in 3NF. (Andresson, 1994) proposed a similar approach. However, like much of the related work, no indication is provided as to whether these methods were actually implemented. (Chiang et al., 1994) proposed a method to extract an EER model from the database extension. It assumes that the database is in 3NF, that there is consistent naming of attributes, that there are no erroneous data values and that inclusion dependencies are available. A prototype implementation is described but no results of ufilising the prototype are provided. (Signore et al., 1994) describe an expert system prototype for rebuilding a conceptual schema from a logical schema. Unlike ITOC, the approach does not utilise the database extension, so it does not verify the results on the inferencing process. (Rugaber and Doddapaneni, 1993) attempted to automate the extraction of and SQL schema from a COBOL program that used fiat ISAM files. Like ITOC, the recovered database design information was loaded into a CASE tool prior to forward engineering into SQL. Unlike ITOC, however, the structure of the application programs are not recovered. Other researchers, such as (Navathe and Awong, 1988), (Markowitz and Makowski, 1990), (Winans and Davis, 1990), (Fong and Ho, 1993), (Premerlani and Blaha, 1994), (Campbell and Halpin, 1996), and (Lim and Harrison, 1996), have described similar approaches for extracting conceptual models from either relational, hierarchical or network data models. The next class of related work we review attempts to migrate the application programs as well as just the static schema. (Zoufaly et al, 1995) describe an emulation approach for migrating RPG to Oracle PL/SQL and Forms 3. Low-level transformation, implemented using their own highlevel language, Logistica are used to directly translate RPG statements to Forms 3. Although an implementation is reported, the target system's representation is at the same low level of abstraction as the RPG source which does not facilitate understanding and maintenance. COBOL/SRE (Engberts, et al., 1993) is described as a "renovation and reengineering" environment. The environment produces reports and support
55 sophisticated browsing and enhancement to an existing COBOL system. Unlike ITOC, this tool only supports forward engineering back into COBOL and so does not raise the level of abstraction. No results were presented as to the effectiveness of the environment. (Karakostas, 1992) describes a similar system. (Vermeer and Apers, 1995) described an approach for recovering design information from 3GL application programs based on the observation that "C" data structures that store the result of SQL queries can serve as object schemata. No indication is given that a prototype has been constructed to evaluate the feasibility of the approach. (Hainaut et al., 1995), proposed a design recovery methodology and generic architecture for performing the extraction of an enhanced ER schema from a database followed by the conceptualisation of the resulting data structures. The ITOC tool described in this paper is novel in that it recovers the structure of 4GL applications at a high level so that new application programs can be generated using a CASE tool. It is also unusual because it has been applied to commercial legacy information systems and the results evaluated by experienced information system developers.
6. Conclusion In this paper, we have outlined the functionality of the ITOC tool and described the results obtained from applying the tool to commercial deployed legacy information systems. The experiment conducted indicates that the approach can be successfully used to recover certain fundamental design components, such as module data diagrams that capture the semantics of the information system application, and also database constraints that are not explicit in the data definition language. The experimentation revealed that "industrial-strength" tools can be constructed using the approach as both a contemporary commercial CASE repository product and (deployed) commercial information systems implemented using a contemporary, commercial 4GL were utilised. As we expected, there were some design constructs that the tool should have recovered from the source application but failed to do so. However, it is likely that these deficiences can be corrected and do not represent a fundamental flaw in the approach. Consequently, we encourage other researchers to utilise it. Our experience indicates that it is essential to apply software re-engineering tools on deployed, commercial applications to demonstrate their true effectiveness. It is also beneficial to have the output of the tools reviewed by experienced practitioners to gain useful feedback for tool improvement. From this feedback, we learned that the ITOC approach must be improved in the area of business rule recovery. Ongoing research involves the monitoring of practitioners who are utilising the ITOC tool to determine what proportion of the total re-engineering effort is eliminated due to its use. We are also investigating metrics that can be used to estimate this effort, which will be extracted automatically from a source application via a utility that includes the source 4GL language model. Finally, we also plan to study the end-user's acceptance of the re-engineered systems produced using the ITOC tool and the costs of retraining these users.
56
Acknowledgements: The authors wish to recognize the contributions of Professor Paul Bailes, Dr. Anthony Berglas and Mr. Ian Peake (CSM), and also Mr. Blair Layton and Mr. Ronald Bradford (Oracle Corporation).
References Andersson, M., Extracting an Entity Relationship Schema from a Relational Database through Reverse Engineering. Proc. of the 13th Entity-Relationship Conference, Manchester, UK, Dec. 1994, pp. 403-419. Berglas A. and Harrison, J.V., Evaluation of the ITOC Information System Design Recovery Tool, Proc. of Fifth International Workshop on Program Comprehension, Michigan, March 1997, pp 176-182. Blaha, M.R. and Premerlani, W. J., Observed Idiosyncracies of Relational Database Designs, Proc. of Second Working Conference on Reverse Engineering, Toronto, Ontario, July 1995, pp. 116-125. Callahan, D., The Program Summary Graph and Flow-sensitive Interprocedural Data Flow Analysis, Proc. of the SIGPLAN'88, Conference on Programming Language Design and Implementation, Atlanta, Georgia, June 1998, pp. 22-24. Campbell, L.J. and Halpin, T.A., The Reverse Engineering of Relational Databases. Proceedings of the 5th Workshop on the Next Generation of CASE Tools, Utrecht, June 1994, pp 50-66. Chiang, H.L., Barton, Terrence M., and Storey, V.C., Reverse engineering of relational databases: Extraction of an EER model from a relational database. Data & Knowledge Engineering 12 (1994), pp. 107-142. Elmasri R. and Navathe S.B., Fundamentals of Database Systems. 2nd Ed. The Benjamin/Cummings Publishing Comp, Inc. 1994. Engberts, Andre, Kozaczynski, Liongosari, Edy and Ning, J.Q., COBOL/SRE: A COBOL System Renovation Environment, Proc. of the 6th Intl. Workshop for Computer-Aided Software Engineering, Ed. H. Lee and Thonas Reid, Singapore, July 1993, pp. 199-210. Fong, J. and Ho, M., Knowledge-based approach for abstracting hierarchical and network schema semantics. Proc. of the 12th Entity Relational Approach, Arlington, Texas, Dec. 1993, pp. 508-519. Harrison, J.V., Bailes P.A., Berglas A., Peak I., Re-engineering 4GL-based Information System Applications. Proc. of the Asia Pacific Software Engineering Conference, Brisbane, Dec. 1995, pp. 448-457. Harrison, J.V. and Berglas A., Data Flow Analysis with the ITOC Information System Design Recovery Tool, Proc. of Automated Software Engineering Conference, Incline Village, Nevada, USA, Nov. 1997, pp.. Harrison, J.V., Bailes P.A., Berglas A., Peak I., Legacy 4GL Application Migration Via Knowledge-Based Software Engineering Technology: A Case Study, Proc. of Australian Software Engineering Conference, Sydney, Oct. 1997, pp. 70-78. Hainaut, J.L., Englebert, V., Henrard, J., Hick, J.M., Roland, D., Requirements for Information System Reverse Engineering Support, Proc. of the 2nd Working Conference on Reverse Engineering, Toronto, Ontario, Canada, July 1995, pp 136-145.
57 Lim, W.M. and Harrison, J.V., An Integrated Database Re-engineering Architecture A Generic Approach, Proc. of Australian Software Engineering Conference, Melbourne, Aug. 1996, pp. 146-154. Karakostas, V., Intelligent Search and Acquisition of Business Knowledge from Program. Software Maintenance: Research and Practice, Vol. 4, (1992), pp. 1-17. Markowitz, V.M. and Makowsky, J. A., Identifying Extended Entity-Relationship Object Structures in Relational Schemas. IEEE Transactions on Software Engineering. Vol. 16, No. 8. Aug. 1990, pp. 777-790. Marlowe, T.J., and Ryder, B.G., 1990. An efficient hybrid algorithm for incremental data flow analysis. In Conf. Recored of the 17 th Annual ACM Symp. On Principles of Programming Languages (San Francisco, Jan.), ACM Press, pp. 184-196. Navathe, S.B. and Awong, A.M., Abstracting Relational And Hierarchical Data With A Semantic Data Model. Proceedings of 8th Entity Relational Approach: A bridge to the future, New York, Nov. 1988, pp.305-333. Peak, I.., User's Guide to the Language Extension Work Bench Version 1, Technical Report 387, Centre for Software Maintenance, Department of Computer Science, University of Queensland, Australia, March 1996. Petit, J.M., Toumani, F., Boulicaut, J.F., Kouloumdjian, J., Using Queries to Improve Database Reverse Engineering. Proc. of the 13th Intl. Conf. on Entity Relational Approach, Manchester, Springer-Verlag, 1994, pp. 369-386. Premerlani, W.J. and Blaha, M.R., An Approach for Reverse Engineering of Relational Databases, Communication of the ACM, Vol.37, No.5 May 1994, pp. 42-49. Rugaber, S. and Doddapaneni, S., The Transition of Application Programs from COBOL to a Fourth Generation Language, In Proc. of Intl. Conference on Software Maintenance, 1993, pp. 61-70. Ryder, B.G., Landi, W., and Pande, H. Profiling an incremental data flow analysis algorithm. IEEE Trans. Software Eng., 16, 2: 1990, pp. 129-140. Signore, Oreste, Loffredo, Mario, Gregori, M., and Cima Marco, Reconstruction of ER Schema from Database Applications: a Cognitive Approach, Proc. of the 13th Intl. Conf. on Entity Relational Approach, Manchaster, Springer-Verlag, 1994, pp. 387-402. Technical Report on the Ingres to Oracle Design Recovery Workbench, Department of Computer Science and Electrical Engineering, University of Queensland, 1996. Vermeer, M.W.W. and Apers, P.M.G., Reverse engineering of relational database applications. Proc. of the 14th International Conference on Object-Oriented Entity Relationship Modelling, Gold Coast, Australia, Dec. 1995, pp. 89-100. Winans, J., and Davis K.H., Software Reverse Engineering from a Currently Existing IMS Database to an Entity-Relationship Model, Proceedings of 9'h Entity Relationship Approach, Lausanne, Switzerland, Oct. 1990, pp. 333-348. Zoufaly, Federico, Araya, Carlos, Sanabria I. and Bendeck, F., RESCUE: Legacy System Translator, In Proc. of the Second Working Conference on Reverse Engineering, Toronto, Ontario, July 1995, pp 39-50.
Adapting Function Points to Object Oriented Information Systems* G. Antoniol 1, F. Calzolari 1, L. Cristoforetti 1, R. Fiutem 1 and G. Caldiera 2 1 I.T.C.-I.R.S.T., Via alla Cascata, 1-38050 Povo (Trento), Italy tel. +39 461 314-444 e-mail : antoniol, calzolar, cristofo, fiutem@irst, itc. it
University of Maryland, Dept. of Computer Science, College Park, Maryland 20742, USA tel. § 301 405-2707 e-mail: [email protected]
Abstract. The object oriented paradigm has become widely used to develop large information systems. This paper presents a method for estimating the size and effort of developing object oriented software. The approach is analogous to function points, and it is based on counting rules that pick up the elements in a static object model and combine them in order to produce a composite measure. Rules are proposed for counting "Object Oriented Function Points" from an object model, and several questions are identified for empirical research. A key aspect of this method is its flexibility. An organization can experiment with different counting policies, to find the most accurate predictors of size, effort, etc. in its environment. "Object Oriented Function Points" counting has been implemented in a J a ~ tool, and results on size estimation obtained from a pilot project with an industrial partner are encouraging. Keywords: Object oriented design metrics, function points, size estimation.
1
Introduction
C o s t a n d effort e s t i m a t i o n is an i m p o r t a n t a s p e c t o f t h e m a n a g e m e n t o f s o f t w a r e d e v e l o p m e n t p r o j e c t s and it c o u l d b e a c r i t i c a l p o i n t for c o m p l e x i n f o r m a t i o n s y s t e m s . E x p e r i e n c e shows h o w difficult is t o p r o v i d e a n a c c u r a t e e s t i m a t i o n : in l i t e r a t u r e 18 a n a v e r a g e e r r o r o f 100% is c o n s i d e r e d t o b e "good" a n d an average e r r o r o f 32% t o b e " o u t s t a n d i n g " . M o s t r e s e a r c h on e s t i m a t i n g size a n d effort has d e a l t w i t h t r a d i t i o n a l a p p l i c a t i o n s a n d t r a d i t i o n a l software d e v e l o p m e n t p r a c t i c e s , w h i l e f e w w o r k s h a v e b e e n e x p e r i m e n t e d for o b j e c t o r i e n t e d ( O O ) s o f t w a r e d e v e l opment. This research was funded by SODALIA Spa, Trento, Italy under Contract n. 346 between SODALIA and Istituto Trentino di Cultura, Trento, Italy.
60 This paper presents a method for estimating the size and development effort of object oriented software, supported by a tool, implemented in Java. The proposed approach, that we call "Object Oriented Function Points" (OOFP), is based on an adaptation for object oriented paradigm of the classical Function Point (FP) methodology 2. As shown in Figure 1, we will measure Object Oriented Function Points, and correlate them with actual system size and development effort to identify estimation models tailored for a specific environment. One of the advantages of this approach is t h a t different estimation models can be developed for different stages of a software project, as soon as the software artifact becomes more detailed while the project goes on. The OOFP_Counter, the Java tool that implements the proposed approach, provides a way to finely tune the counting rules by setting several parameters related to which counting policy is better suited for a given software project. This paper is organized as follows: Section 2 explains how we map main concepts of function points to object oriented software. The rules for counting Object Oriented Function Points are then described in Section 3, with emphasis on different counting policies t h a t can be adopted. Section 4 presents the OOFP_Counter, the tool developed to automatize the counting process. This tool has been used to produce results for an industrial pilot project, focused on size estimation, reported in Section 5. Finally, conclusions are drawn.
2
Object Oriented Function Points
Since they have been proposed in 1979 1, function points (FP) have become a well known and widely used software metric. Despite some concerns 10, 11, 12, 17, practitioners have found FPs to be useful in the data processing domain, for which they were invented. Function points are available at the specification phase since they are based on the user's view of software functionality. FPs are generally considered to be independent from the technology used to implement the solution. The key features of function points are t h a t they are available early, and they are a measure of the problem independent from any particular implementation. The International Function Point Users Group (IFPUG) publishes guidelines to standardize their definition 6.
Fig. 1. Measures in the software development process. Several variants have been proposed to extend FPs use to other domains (see 5 for a survey). Since OO paradigm had become widely adopted to design large information systems, different attempts have been proposed to adapt function points concepts to object oriented software, in order to exploit the understanding gained with function points in their traditional domain. In the object oriented approach, an object model uses classes and their inter-relationships to represent the structure of a system. While the development proceeds the object model evolves: in addition to the problemrelated classes, the model includes design- and implementation-oriented classes with new inheritance relationships. These changes do not concern the user, but reflects the developer's view of the system. A measure derived from the object model should be now a better predictor of development size and effort. The OOFP approach enables a smooth transition from the user's view to the developer's view, and the same methodology can be used to measure the object model at each stage, as shown in Figure 1.
2.1
Mapping
function points to object oriented
software
Object model, dynamic model, and functional model may be used to represent information about object oriented software 14. The object model is usually the first to be developed, and it is the only one that describes the system using specifically object-oriented concepts. We focus our attention to object model to map traditional FP concepts to OOFP, translating logical files and transactions to classes and methods. A Logical File (LF) in the function point approach is a collection of related user identifiable data. Since a class encapsulates a collection of data items,
62 it seems to be the natural candidate for mapping logical files into the OO paradigm. Objects that are instances of a class in the OO world correspond to records of a logical file in data processing applications. In the FP method the application boundary identifies Internal Logical Files (ILFs) (logical files maintained by the application) and External Interface Files (EIFs) (referenced by the application but maintained by other applications). In the 0 0 counterpart, we could consider external classes encapsulating non-system components, such as other applications, external services, and library functions. Classes within the application boundary correspond to ILFs. Classes outside the application boundary correspond to EIFs. In the OO paradigm operations are performed by methods (which are usually at a more fine-grained level than transactions). Since object models rarely contain the information needed to tell whether a method performs an input or an output or is dealing with an enquiry, we simply treat them as generic Service Requests (SRs), issued by objects to other objects to delegate some operations. Issues such as inheritance and polymorphism affect the structure of the object model, and how the model should be counted. This problem will be addressed in Section 3.1. 2.2
Related work
Several authors have proposed methods for adapting function points to object oriented software. In 15 classes are treated as files, and services delivered by objects to clients as transactions, while in 19 each class is considered as an internal file, and messages sent across the system boundary are treated as transactions. Sneed 16 proposed object points as a measure of size for OO software. Object points are derived from the class structures, the messages and the processes or use cases, weighted by complexity adjustment factors. A draft proposal by IFPUG 7 treats classes as files, and methods as transactions. Fetcke 3 defines rules for mapping a "use case" model 9 to concepts from the IFPUG Counting Practices manual, but no attempt has been made to relate the results to other metrics, such as traditional function points, lines of code, or effort. The key aspect of our approach is its flexibility. For example, Fetcke 3 defines that aggregation and inheritance should be handled in a particular way. We define several options (one of which is Fetcke's approach) and leave it to the user to experiment which parameter settings produce the most accurate predictors of size, effort, etc. in its environment. Thus we have a method which can be tailored to different organizations or
63
environments. Moreover, the measurement is not affected by subjective ratings of complexity factors, like those introduced in classical function point analysis. Finally, the OOFP_Counterwill automatically count OOFPs, for a given setting of parameters.
3
Measurement Process
OOFPs are assumed to be a function of objects comprised in a given object model D (D can be that produced at design stage or extracted from the source code) and they can be calculated as:
OOFP = OOFPILF + OOFPEIF -b OOFPsR where:
OOFPxLF ----~
WILF(DETo, RETo)
oEA
OOFPEIF = ~ W~LF(DETo, RETo) of~A OOFPsn = ~
Wsn(DETo, FTRo)
oEA
A denotes the set of objects belonging to the application considered and o is a generic object in D. Dets, Rets and Ftrs are elementary measures to be calculated on LFs and SRs and used to determine their complexity
In 001~ i
I
Fig. 2. OOFP computation process.
64
through the complexity matrixes W. Such meaasures are further detailed in Sections 3.2 and 3.3. Counting OOFPs is a four steps process: 1. The object model is analyzed to identify the units that are to be counted as logical files. 2. The complexity of each logical file and service request is determined. Structural items are mapped to complexity levels of low, average, or high. 3. The complexity scores are translated into values. 4. The values are summed to produce the final O O F P result. Figure 2 outlines the counting process. The counting rules used in these steps are described in Sections 3.1 to 3.3, while Section 4.1 explores the effect of counting classes in different ways.
3.1
I d e n t i f y i n g l o g i c a l files
Classes are generally mapped into logical files. However, relationships between classes (aggregations and generalization/specializations in particular) can sometimes require to count a group of classes as a single logical file. Different choices of how to deal with aggregations and generalization/specialization relationships lead to different ways to identify logical files. In what follows we are going to present the four different choices we identified: a simple example taken from 4 will support explanation. 1. S i n g l e Class: count each separate class as a logical file, regardless of its aggregation and inheritance relationships (Figure 3). 2. A g g r e g a t i o n s : count an entire aggregation structure as a single logical file, recursively joining lower level aggregations (Figure 4). 3. G e n e r a l i z a t i o n / S p e c l a l l z a t i o n : given an inheritance hierarchy, consider as a different logical file the collection of classes comprised in the entire path from the root superclass to each leaf subclass, i.e. inheritance hierarchies are merged down to the leaves of the hierarchy (Figure 5). 4. M i x e d : combination of option 2 and 3 (Figure 6). Merging superclasses into subclasses makes intuitive sense. It seems right to count leaf classes, with their full inherited structure, since this is how they are instantiated. Dividing a user-identifiable class into an aggregation of sub-classes is an implementation choice. Thus from the point of view of the function point
65
Fig. 3. Single class ILFs.
Fig. 4. Aggregations ILFs. measurement philosophy, the O O F P value should not be affected. From this perspective, the aggregation structure should be merged into a single class and counted as a single logical file. Merging aggregations or not seems to depend on whether the user's or designer's perspective is chosen. However, a hybrid solution can be adopted as well, flagging on the design which aggregations must be considered as a unique entity and thus must be merged.
3.2
Complexity
of Logical Files
For each logical file it is necessary to compute the n u m b e r of DETs (Data Element Types) and RETs (Record Element Types). Counting rules depend on whether it is a simple logical file, corresponding to a single class, or a composite logical file, corresponding to a set of classes.
66
For simple logical files: - One R E T is counted for the logical file as a whole, because it represents a "user recognizable group of logically related d a t a " 6.
F i g . 5. Generalization/Specialization ILFs.
Fig. 6. Mixed ILFs. - Simple attributes, such as integers and strings, are considered as D E T s , as they are a "unique user recognizable, non-recursive field of t h e ILF or EIF" 6. - C o m p l e x attributes are counted as RETs. A complex a t t r i b u t e is one whose type is a class (i.e. "a user recognizable subgroup of d a t a elements within an ILF or EIF" 6) or a reference to a n o t h e r class. A single-valued association is considered as a D E T ( I F P U G suggests counting a D E T for each piece of d a t a t h a t exists because t h e user requires a relationship with another ILF or E I F to be maintained6). - A multiple-valued association is considered as a R E T , because an entire group of references to objects is m a i n t a i n e d in one attribute. Aggregations are treated simply as associations. -
-
67 For composite logical files: - Using the rules for simple logical files, except for the handling of aggregations, DETs and RETs are counted separately for each class within the composite. - In a composite logical file aggregations represent a subgroup. One RET, assigned to the container class, is counted for each aggregation, whatever its cardinality. One more RET is also counted for the logical file as a whole. - The individual DETs and RETs are summed to give an overall total for the composite logical file. W h e n the DETs and RETs of a logical file have been counted, tables (derived from those given in the IFPUG Counting Practices Manual Release 4.0 6 for ILFs and EIFs) are used to classify it as having low, average, or high complexity.
3.3
Complexity
of Service Requests
Each method in each class is considered: abstract methods are not counted. while concrete methods are only counted once (in the class in which they are declared), even if they are inherited by several subclasses. If a method is to be counted, the data types referenced in it are classified as simple items (analogous to DETs in traditional function points) for simple data items referenced as arguments of the method, and complex items (analogous to File Types Referenced (FTRs) in traditional function points) for complex arguments 2. Again tables axe used to classify the method as having low, average, or high complexity. Notice that sometimes the signature of the method provides the only information on DETs and FTRs. In such a case, the method is assumed to have average complexity.
3.4
An
Example
The counting procedure for each individual class gives the DETs and RETs shown in Fignre 7, while Table 1 shows ILF and SR contribution to OOFP counting. Since service requests (methods) are only counted once, it does not matter how the classes are aggregated into logical files. Because the signatures are unknown for the methods in the example, each method is assumed to have average complex_ity.
68 MapSite
DET=I RET=I
Ent~ '
I
-'~ Room ~ roomNumbcr | Enter | SetSide I C~tsia~ DET=O RET=2
DET=2 RET=2
DET--O RET=I
D ET =I RET=I
Fig. 7. D E T / R E T computation for LFs on the example system.
Values in third and fifth columns show the results of applying IFPUG 4.0 complexity tables with each variant. The value 7 is r a t e d as Low and it is weighted 4. For more details about how counting rules have been applied the interested reader could refer to 2.
Single Class Aggregation Generalization/Specialization Mixed
ILF ILF OOFP 5 35 4 28 4 28 3 21
SR SR OOFP Total OOFF 7 28 63 7 28 56 7 28 56 7 28 49
Table 1. ILF and SR complexity contribution.
The highest OOFP count comes when each class is counted as a single ILF. All the other variants have the effect of reducing the O O F P value, as they reduce the number of ILFs. Although there is an increase in D E T s / R E T s in the merged ILFs, it is not enough to raise the ILF complexity to higher values. For this example, and for the pilot project that will be presented in Section 5, the complexity of each ILF and SR are always determined to be low. The tables used to determine complexity are based on those from the I F P U G Counting Practices Manual 6, in which quite large numbers of R E T s and DETs are needed to reach average or high complexit3" (for example, to obtain an average complexity weight an ILF needs a D E T value between 20 and 50 and a RET value between 2 and 5). On the d a t a
69
available to us so far, we suspect that recalibration of the O O F P tables for logical files might improve the accuracy of OOFP as a predictor of size, but further experimentation is needed on this topic.
4
T h e O O F P _ C o u n t e r Tool
We have developed the OOFP_Counter tool, presented in Figure 8, to automate the OOFP counting process. This tool has been implemented using Java. The OOFP_Counter inputs Abstract Object Language (AOL) specification of the object oriented model. AOL is a general-purpose design description language capable of expressing concepts of OO design. It has been adopted in order to keep the tool independent of the specific CASE tool used. AOL is based on the Unified Modeling Language 13, which represents de facto a standard in object oriented design. The OOFP_Counter tool parses AOL specification and produces an abstract syntax tree representing the object model. The parser also resolves references to identifiers, and performs some simple consistency checking (e.g. names referenced in associations have been defined). To improve portability, the AOL parser and the O O F P counter, the two parts of the OOFP_Counter tool have been implemented in Jax~a. For the project presented in Section 5, OMT/STP 8 has been used as CASE tool; an automatic translator to convert from O M T / S T P output to AOL specifications has been implemented.
i
oo,
OCFP Coun~r
Fig. 8. The OOFP_Counter tool.
70 4.1
Parameters
Setting
The OOFP_Counter works on the abstract syntax tree and implements the OOFP Counting Rules described in section 3. It is possible to set several parameters, that may influence the counting policy: ILF counting strategy (see Section 3.1) External classes inclusion - Private methods counting; Private attributes counting; - Values of DET, RET, and FTP~ thresholds between low, average, and high complexity. -
Parameter setting might be guided by some philosophy. For example, from a traditional function point perspective one would wish to count only user-visible abstractions, ignoring all implementation aspects. This might mean selecting the Mixed strategy for grouping classes into logical files, counting only those methods which are publicly visible and related to classes at the system boundary, and giving full weight to classes whether they are reused or not. From a designer's point of view, one might want to take account of all implementation details, in an attempt to get an accurate estimate of development effort. This might mean counting each class as a separate logical file, including all methods and attributes, and reducing the weight given to reused classes. Different parameter settings could be tried on a purely experimental basis in order to identify that company specific profile that gives the best overall performance for estimating size or effort.
5
An Industrial Case Study
The described methodology has been applied in an industrial environment. Our first study is of the relationship between the O O F P measure of a system and its final size in lines of code (LOC), measured as the number of non-blank lines, including comments. Size estimation is important, since it is needed for most effort estimation models, thus we can make use of existing models that relate size to effort. Eight completed (sub-)systems were measured, for which both an OO design model and the final code were available. All were developed in the same environment, using the C + + language. Table 2 shows the size of each system, spreading from about 5,000 to 50,000 lines of code.
71
Table 2 also shows the OOFP count for each system, using each of the four different strategies for identifying logical files.
System LOC Single Class Aggregation Generalization Mixed (SC) (AB) (GB) (MB) A 5089 63 63 35 35 B 6121 476 462 455 469 C 15031 284 284 270 270 D 16182 1071 1057 1057 1043 E 21335 562 513 548 499 F 31O11 518 403 483 368 G 42044 1142 1100 1124 1072 H 52505 2093 1947 1872 1737 T a b l e 2. System sizes and O O F P s . T h e four O O F P series are strongly correlated each other, with all correlations within the .992 - . 9 9 8 range (Pearson), the lowest corresponding to SC vs MB. As shown in Table 2, differences between t h e m e t h o d s become appreciable only for the projects with large L O C values. Several regression techniques were considered to model t h e L O C - O O F P association. Given the reduced size of the database, a leave-one-out crossvalidation procedure was used to achieve unbiased estimates of predictive accuracy for the different models. Model error was expressed in t e r m s of n o r m a l i z e d m e a n squared error (NMSE): each model was trained on n - 1 points of the data base L (sample size is currently n = 8) a n d tested on t h e withheld datum; NMSE is obtained over L normalizing over the sample variance of the observed values (#~ = m e a n ( y ) ) . T h e small size of the database and a limited knowledge of LOC measures validity required the use of simple models capable to handle non obvious outliers in the response variable LOC. In this study, t h e basic least squares linear fit was compared with resistant techniques. Regression estimates based on least square minimization are in fact sensitive to outliers in the response variable when the error distribution is not Gaussian. Robust regression techniques may improve t h e least-squares fit and handle model inadequacies due to unusual observations. First linear models (1ms) based on the minimization of the s u m of squares of the residuals were developed for each ILF selection m e t h o d . Least absolute deviation, based on L1 error was also applied (11s) . T h e regressor is build minimizing the sum of the absolute values of the residuals to resist the effect of large error values.
72
Method N M S E N M A E lm-SC 0.391 0.661 lm-SC-1 0.539 0.811 lm-AB 0.434 0.656 lm-GB 0.380 0.601 lm-MB 0.464 0.681
/~2 0.730 0.901 0.691 0.728 0.680
bo 7992.5 0000.0 8504.7 7435.1 8187.4
bl 23.0 29.4 23.8 25.2 25.8
ll-SC ll-AB ll-GB ll-MB
0.547 0.629 0.389 0.457
0.812 0.855 0.693 0.734
-
9139.1 8601.1 8688.4 8083.0
21.58 23.48 24.36 26.61
rreg-SC rreg-AB rreg-GB rreg-MB
0.399 0.431 0.368 0.443
0.672 0.661 0.599 0.664
-
7875.2 8255.3 7331.7 7861.9
23.0 24.0 25.5 26.4
rlm-SC rlm-SC-1 rlm-AB rlm-GB rlm-MB
0.402 0.633 0.440 0.377 0.456
0.670 0.860 0.660 0.600 0.676
-
8001.9 0000.0 8517.5 7521.5 8161.6
23.0 29.3 23.8 25.6 26.3
Table 3. Model performance for linear regressors (lms and lls) and robustified methods (rregs and rhns). The normalized mean squared error (NMSE) and the normalized mean absolute error (NMAE) are estimated by cross-validation.
A family of M-estimators was therefore considered ( r r e g s and r l m s ) . The basic idea of M-smoothers is to control t h e influence of outliers by the use of a non-quadratic local loss f u n c t i o n which gives less weight to "extreme" observations. Non-linear modelling was also a t t e m p t e d , expecting instability and lack of convergence due to t h e sample size. E s t i m a t e d model accuracy for each model ~ = bo + b l x of each experimented family is collected in Table 3, p a r a m e t r i z e d over I L F selection m e t h o d s and type of regressor. The model coefficients bo a n d bl are indicated as computed from the full d a t a set. E s t i m a t e d R - s q u a r e d measure is also included for the linear models for comparison w i t h other results separately obtained on these data.
73
A point of concern is the inclusion of an intercept t e r m bo in model: it is reasonable to suppose the existence of s u p p o r t code unreferred to Method rreg-default-GB rreg-andrews-GB rreg-bisquare-GB rreg-fair-GB rreg-hampel-GB irreg-huber-GB rreg-logistic-GB rreg-loglstlc-GB-0.8 rreg-talworth-GB rreg-welsch-GB
Comments converged after 50 steps) c = 1.25 c = 0.80 -
Table 4. Model performances for different weighting functions of the M-estimator rreg. Results are given for the GB selection method only.
t h e functionalities being counted, and prediction is improved whith the term. However, the intercept term is not significant in a non-predictive fit of t h e data. More important, the fact that the intercept t e r m is alw~.s larger t h a n the first LOC value might indicate poor fit for small O O F P values. It would be interesting to apply a Bayesian procedure to select t h e intercept from given priors. T h e estimates for different weighting functions of the M-estimator are listed in Table 4. T h e best predictive accuracy (NMSE= 0.337) was achieved by the rreglogistic-GB model with tuning parameter u -- .8, corresponding to t h e linear predictor L O C --" 7183.4 + 25.6 GB. As shown in Figure 9, the rreg-logistic-GB model is very close to t h e basic linear model lm-GB, whose equation is L O C = 7435.1 + 25.2 G B . As the GB m e t h o d is consistently better for all models a n d for b o t h t h e predictive error measures NMSE and NMAE, these results indicate t h a t t h e choice of ILF selection method may influence prediction. Lowess, s u p e r s m o o t h e r and predictive splines have been also tested a n d showed instability of convergence due to the small sample size. A l t h o u g h more experimental work is needed, obtained results are encouraging for size estimation.
74
6
Conclusions
This paper shows how the concepts of function points can be applied to object oriented software. We presented a methodology for estimating the size and effort of object oriented software. The method is based on an adaptation of traditional function points to object oriented paradigm. Mapping from F P concepts to OO concepts have been defined, and the O O F P s counting process
LOC = 7163.4+ 25.6 GB
i 0
i 500
j
t 1000
I
J 1500
GB
Fig. 9. The rreg-logistic-GB model (c=0.8) compared with the linear model lm-GB.
has been described. The OOFP_Counter tools has been developed to automate the counting process. Results obtained from a pilot study in an industrial environment have been reported. The results for size estimation are encouraging, a n d t h e y can be used with many effort estimation models. Future work will investigate the effect of recalibrating t h e complexity tables and analyzing the statistical correlation between t h e collected measeres (DETs, RETs, FTRs) and program size. O t h e r relationships, beyond just OOFP and code size, will be studied; those between O O F P and traditional FP, and O O F P versus effort, are of particular interest.
75
7' A c k n o w l e d g e m e n t The authors are indebted with Cesare Furlanello who performed most of the statistical analysis in the pilot study.
References 1. A. J. Albrecht. Measuring application development productivity. In Proc. IBM Applications Development Symposium, pages 83-92. IBM, Oct. 1979. 2. G. Caldiera, C. Lokan, G. Antoniol, R. Fiutem, S. Curtis, G.L. Commare, and E. Mambella. Estimating Size and Effort for Object Oriented Systems. In Proc. ~th Australian Conference on Software Metrics, 1997. 3. T. Fetcke, A. Abran, and T.-H. Nguyen. Mapping the OO-Jacobson approach to function point analysis. In Proc. IFPUG lg97 Spring Conference, pages 134-142. IFPUG, Apr. 1997. 4. E. Gamma, P~. Helm, 1~. Johnson, and J. Vlissides. Design Patterns: Elements of Reusable Object Oriented Software. Addison-Wesley, 1995. 5. T. Hastings. Adapting function points to contemporary software systems: A review of proposals. In Proc. 2nd Australian Conference on Software Metrics. Australian Software Metrics Association, 1995. 6. IFPUG. Function Point Counting Practices Manual, Release ~.0. International Function Point Users Group, Westerville, Ohio, 1994. 7. IFPUG. Function Point Counting Practices: Case Study 3 - ObjectOriented Analysis, Object-Oriented Design (Draft). International Function Point Users Group, Westerville, Ohio, 1995. 8. Interactive Development Environments. Software Through Pictures manuals, 1996. 9. I. Jacobson, M. Christerson, P. Jonsson, and G. C)vergaard. Object Oriented Software Engineering: A Use Case Driven Approach. Addison-Wesley, 1992. 10. D. Jeffery and J. Stathis. Function point sizing: Structure, validity and applicability. Empirical Software Engineering, 1(1):11-30, 1996. 11. B. Kitchenham and K. K~ins/il~. Inter-item correlations among function points. In Proc. 15th International Conference on Software Engineering, pages 477-480. IEEE, May 1993. 12. B. Kitchenham, S. Pfleeger, and N. Fenton. Towards a framework for software measurement validation. IEEE Transactions on Software Engineering, 21(12):929-944, Dec. 1995.
76 13. Rational Software Corporation. Unified Modeling Language, Version 1.0, 1997. 14. J. Rumbaugh, M. Blaha, W. Premerlani, F. Eddy, and W. Lorensen. Object Oriented Modelling and Design. Prentice-Hall, 1991. 15. M. Schooneveldt. Measuring the size of object oriented systems. In Proc. 2nd Australian Conference on Software Metrics. Australian Software Metrics Association, 1995. 16. H. Sneed. Estimating the Costs of Object-Oriented Software. In Proceedings of Software Cost Estimation Seminar, 1995. 17. J. Verner, G. Tate, B. Jackson, and R. Hayward. Technology dependence in Function Point Analysis: a case study and critical review. In Proc. 11th International Conference on Software Engineering, pages 375-382. IEEE, 1989. 18. S. Vicinanza, T. Mukhopadhyay, and M. Prietula. Software-effort estimation: an exploratory study of expert performance. Information Systems Research, 2(4):243-262, Dec. 1991. 19. S. Whitmire. Applying function points to object-oriented software models. In Software Engineering Productivity Handbook, pages 229244. McGraw-Hill, 1993.
Global Cache Management for Multi-class Workloads in Data Warehouses Shudong jinl, and Xiaowei Sun2 Department of Computer Science, Huazhong University of Science & Technology, Wuhan, Hubei 430074, China [email protected] 2 College of Computer Science, Northeastern University, Boston, MA 02115, USA [email protected]
Data warehouses usually rely on the underlying database management systems, and operations in warehouses require adequate global cache management for both base data and derived sets. However, a problem is how to ensure the cache balance between the query operations and the retrieved sets of query templates. In this paper, we describe an efficient global cache manager that caches both database pages and derived data. A benefit metric is developed to compare the expected effect of caching retrieved sets and buffering for query processing. On the basis of this model, algorithms are proposed to generate efficient cache allocation scheme for multiple queries. These algorithms can be combined with different replacement strategies. We designed a mixed workload, and made experiments to investigate the performances of the algorithms. The results indicate that our approaches are better than traditional LRU and LRU-K methods for multi-class workloads in data warehouses. Abstract.
1
Introduction
Warehousing is a promising technique for retrieval and integration of data from distributed, autonomous and possibly heterogeneous information sources 19. Data warehouses are usually dedicated to the online analytical processing (OLAP) and decision support system (DSS) applications. Many advanced techniques have been incorporated in warehouses to gain an acceptable level of performance. Literature focused mainly on materialized view selection 2, 10 and maintenance 9, 12, 17, and query equivalence for using the views 8. Only a few researchers have discussed cache management in warehouses. Design of efficient buffer management algorithms has gained a lot of attention. The LRU-K replacement algorithm 15 uses the last K reference times of each cached object. For multiple-class workloads, DBMIN 3 is a classic method. In 14 a flexible method was proposed. It considers different query access patterns and uses adaptable replacement strategies. Recently, Brown et al. revisited this problem 1. They developed a new method Class Fencing based on the concept of hit rate concavity.
78 These approaches depend on the uniform size of all pages and the uniform cost of fetching these pages. In data warehouses, the cache criteria should consider different retrieved set sizes, execution costs of associated query templates, as well as their reference rates. In 16, Sellis studied cache replacement algorithms in the context of caching retrieved sets of queries. Several cache replacement strategies were pro-posed, which sort the retrieved sets by one of above three factors or their weighted sum. Another project ADMS 5 benefits from caching at multiple levels, i.e., the retrieved sets and the pointers to the retrieved records. A recent important work was reported in 18. Their cache manager aims at minimizing query response time. Two complementary cache replacement and admission algorithms make use of a profit metric. This metric combines the reference rate and size of each retrieved set, and the associated query execution cost. Experimental results indicated that their algorithms outperform conventional LRU replacement algorithm in decision support environments. Unfortunately, caching methods for multi-class workloads in data warehouses have not been explored sufficiently. Especially there exists a problem: How to cache for both the query operations and the retrieved sets of query templates? When the memory contains derived data, and other operations also ask for buffer space, how to ensure a balance between these two needs and achieve maximum gain? It is an important and interesting theme for several reasons. First, many warehouses are constructed on the top of databases, so it's required to cache both database pages and derived data in a global buffer pool. Second, operations in warehouses can be very complex and multiform: some operations on set data are completed by warehouse software; others may rely on the underlying DBMS and base data. These operations all require memory for processing. Finally, warehouses should support multi-user environments. OLAP/DSS processing and small updates and queries generate mixed workloads. Even a single complex query can be decomposed into several basic operations. In this paper, we investigate global cache management for multi-query workloads in warehousing environments. A practical metric is developed to compare the benefits of caching retrieved sets and buffering for query instances. Algorithms are designed to obtain efficient cache allocation based on this metric. Hence they produce cache allocation schemes considering both needs. These algorithms integrate with different replacement strategies to generate several cache management methods. Simulated results show that our approaches outperform LRU and LRU-K algorithms. The rest of this paper is organized as follows. Section 2 contains a brief introduction to our global cache manager in a warehousing architecture. Section 3 models the profit of caching retrieved sets and the benefit of buffering for query operations, and links them with a practically comparable metric. Then in section 4, we describe the cache management algorithms. In section 5, a mixed workload is designed on an experimental database to study the algorithms' performances, followed by a brief conclusion.
2
Architecture
Data warehouses usually rely on traditional DBMS, not only for the data provided by databases, but also for the soRware construction. For example in warehousing envi-
79 ronments, the buffering subsystem should cache both database and warehouse data in a global buffer pool. Figure 1 gives a simple framework of data warehouse management system (DWMS) on the top of a database management system (DBMS), and a global cache manager (GCM) in it. This architecture is similar to our warehouse research project based on a client-server database system.
Applications DWMS Que~ instances DBMS (3)~ ~(4) I Cache manager
(1)1' I (2) ~es.
,
(1) Cachethe base data (2) Cachethe deriveddata (3) Access and process the base data (4) Access and process the derived data
!nD~r:::det~ Figure 1. Global cache manager in A DWMS
The global cache manager caches database pages and derived data of warehouses. At low level, derived sets are cached in a granule of individual pages, but each page has an identifier to indicate its corresponding set. The cache is organized efficiently by using hashed lists. Each buffer page contains a page identifier, page content, four link pointers (two for the hashed list and two for the global list), some flags, referenced bits (each bit indicates whether the page is referenced by the corresponding thread), and the last K reference times for LRU-K replacement algorithm. Query threads are called query instances; each is dispatched for a basic operation. Multiple users' queries, on either base or derived data, can be processed concurrently. Each might be decomposed into several operations. For example, a query can be decomposed into three instances: One retrieves a derived set; one selects records from a table; and the third joins above two results. They call the cache allocation algorithms to require buffer space. After obtaining cache, they can decide how to utilize it. A query instance's allocated cache is called its private cache. The remains are flee to be dedicated to new instances or swapped out by replacement algorithms.
3
Benefit Model
Warehouses are required to cache the retrieved sets of predefined query templates, as well as to provide working area to query operations. In this section we model the profit of caching derived sets and the effect of buffering for query operations, then solve the problem: How to compare the benefits of satisfying these two needs.
80
3.1 Caching Retrieved Sets Retrieved sets have different accessed frequencies. For example, some small statistical sets (averages, sums, counts, etc.) are usually most frequently referenced, but a multiattribute projection view seems to be seldom used. To cache retrieved sets of frequently cited templates is more efficient, since they are likely to be used again in the near future. We must have some metric to judge the profit of caching them. In buffer management, the usual criterion for deciding to keep which stored objects should consider their reference probabilities in the future. For future reference patterns are absent in advance, the future reference probabilities are approximated from past reference patterns. This method performs well if the patterns are stable. Let DSI, DS2..... DSm be the derived sets of query templates M1, M2..... Mm. We use the following statistics of each derived set DS~ and associated query template M,: F~ : average reference frequency ofM~; S~ : size of DS, (in pages) produced by executing M~; Ct : average execution cost of M,. Let us define the benefit of caching DS,. When DS~ is not in cache and some query references M,, DS, will have to be retrieved. (If DS, is not permanently stored on the disks after the last execution of M,, then the template will have to be recomputed; if DS, is maintained, then the system need only to read the set.) Let C, denote the retrieval cost, which is assumed able to compute. Therefore, if DS~ resides in memory, cost C~ can be saved. We have the following equation to express the average cost saving (it's so expressed since the smaller sets with higher reference rates and retrieval costs have higher profits):
Profit (M, ) = F~x C,
s,
(1)
Obviously, it can benefit from caching the derived sets with high Profit values. To achieve this we should decide the values of F,, C,, and S, for each template M,. As described above, C~ and St can be determined when M, is computed or the set is read from the disks. However, computation of F, is difficult since it depends upon dynamic historical information sampled from processing. As in 18, a similar LRU-K method for set data will be eligible for this work. It records the last K reference times, computes and maintains the reference rate of each retrieved set.
3.2 Caching for Query Instances Queries in data warehouses can be very complex. Some queries are transformed into operations on the derived sets, and others may be directly delivered to the underlying database systems. Even those queries interpreted by warehouse software also depend on basic database operations. A complex warehouse query usually contains several operations on base relations; further processing of retrieved sets is also possible, for example joining two views and aggregating the result. Besides in fact, re-computation of the derived sets is performed by query instances. These operations depend on the
81 underlying routines, including the buffering subsystem. The global cache manager should allocate cache as working areas of multiple queries on different data resources. Consider the problem of finding the optimal cache allocation for multiple queries with maximum buffering effect. The standard to measure the effect of buffer management is the disk I/O times. So we consider the expected page faults of the queries. Let Ql, Q2, ..., Qq be the query instances in the system at some time. A query's access paths determine its reference patterns, and then the number of page faults if buffer size ftxed. Thus the number of Q,'s page fault can be defined as a function PF, (B~), whose only parameter B~ is the buffer number allocated to the query. The objective of cache allocation is to minimize the total page faults YLI PF, (B,) under cache size restriction ~ B ; _
(PF, (B~n ) - PF~(B,)) x tio B, - B ~
(2)
Here B,>Bmin, B n is the minimum cache size that Q, requires necessary for its processing, and tio is the disk I/O cost of a page. If we hope to allot extra b~0 buffer pages to Q~, then this addition's effect is:
Eft(Q,, B,,B, + b,) = (PF,(B,)-PF~(B, + b,))x ti~
(3)
b, According to this equation, we can compute the effects of cache addition to the queries and allocate buffer pages to the one with greatest effect. Similarly in 1, Class Fencing estimates the buffer hit rate by exploiting a hit rate concavity assumption. This assumption says that the slope of hit rate curve never increases as more memory is added to an optimal buffer replacement policy. The slope of hit rate curve represents the marginal increase in hit rate curve obtained by adding an additional buffer page. This theorem supports our measures. A problem in above approach is the derivation of the page fault functions. They rely on the access methods of particular queries, and should be declared or estimated by the query processing routines. We give some examples to illustrate them. Let be the page reference sequence. In relational databases, many complex reference patterns are constructed on the basis of three simple patterns: 9 9
Sequential reference. Its characteristic is, for Vi, j, if l
82
9 Looping reference. There exists a cycle t, for Vi, j, if l
Pir
As described in 6, we can obtain the expected page fault times of above access patterns as follows, respectively. We can also determine the minimum buffer requirements according to MG-x-y 14: P Fscq,~ti~(b) = k Bm~u~
(4)
= 1
(5)
r N x ( 1 - ( N - 1 ) k ) , k < ko = lnO - b / N) PFrandom(b)= .~ N ha( - I/N), k
(6)
Bmin,random = 1
(7)
t§ ( t - b ) x t x ( k / t - 1 ) ?Flooping (b) = (
t- l
t,
b<_t, '
(8)
b>t (9)
Brain,looping = x ~ X t
Here b is the allotted buffer page number, N is the relation size in pages, and x is an adaptable parameter in MG-x-y family.
3.3 Comparable Metric When query instances execute, the cached derived sets may be swapped out due to competition. R becomes serious if many instances run concurrently and require large amount of cache. So it is important to decide whether to cache some derived sets for possible future reference to the query templates, or dedicate the cache to current query instances to reduce their page faults. Unfortunately, equation (1) and (3) provide different metrics to measure the potential benefits of caching derived sets and bufferhag for query instances. It is necessary to develop an approach linking them, thus to compare the benefits of satisfying both needs. We give a practical approximation. Let At be the length of an elapsed period, A/O be the total page I/O operations completed during this period. Assume that Qi has performed I0, page reads, which is monitored and maintained by the cache manager. We obtain the expected query runtime based on historical disk access frequency At/AIO. The expected remaining UO times is ~q=t(PF~(Bi) - I O j ) , and the expected runtime is: Er
6t
= =
AIO
(lO)
83 If DS, will be cached during the future ET period, instead of dedicating this portion of buffer to query instances, then the expected cost saving is: EProfit (M~) = ET • Profit (Mr)
(11)
To decide whether to cache derived sets or to allot buffer to query instances, need only compare the values of equation (3) and (11). The former is used to compute the effect of increasing buffer size for Q,, and the latter estimates that of caching DS~. Above approximated ET may be somewhat inaccurate, but to obtain an accurate ET is considerably complicated, which must consider the processing of all queries. Besides, the approximation also contributes to other factors such as the estimated PF~, and unstable reference patterns, etc. We should further consider a special case in estimating ET. The computation of equation (11) relies on the statistics maintained by the cache manager, but if there is no any query instance in running, then ET can not be computed. If so, when a new query instance Q~ arises, we estimate ET as: ET = PF1 (BI) x EFio
(12)
Here EFio is an approximated I/O fi'equency in processing, 0<-EFio
4
Cache Management Algorithms
In this section, we describe the chief algorithms in the global cache manager. The f'u'st is greedy algorithm GA, which tries to find the best cache allocation scheme (CAS) for a multiple query workload and the derived sets. Other allocation algorithms base upon it. As an example, algorithm CA for allocating cache to new queries is depicted. We also briefly discuss the buffer admission and several possible replacement strategies.
4.1 Cache Allocation Basic Greedy Algorithm The first greedy algorithm (described below) considers allocating total B buffers to the query instances and query templates. Based on the comparable metric described in section 3, it tries to find an adequate scheme, which achieves maximum benefit and ensures cache balance between the derived data and the query instances. The query templates can be regarded as constant queries, so that they and the query instances can be treated in a uniform way. The initial cache allocation scheme is (Bb B2, ..., Bq, DI, s ..... Din). It means that Bj buffer pages are allocated to each Qj, and D~ pages are used to cache the retrieved set of each M,. It always satisfies Eq=~B~ + Y,%ID, ___B , the cache size constraint. D,
84 can be S, or 0, representing whether to intend caching the derived set. But D ~ S , does not mean that DS~ must exactly be cached, and D~=0 does not mean that DS, is not in memory currently. In fact a derived set can be partially cached. Algorithm GA: The greedy algorithm trying to find the best cache allocation scheme. Input: CAS=(B1, B2, ..., Bq,,DI, D2, 7" Din); Output: new CAS=(B'I, B2 .... , Bq, D1, D'2 ..... D'm), in which B'~>-Bi, D',=S, or 0, Eq=, B; + Z,.I D, ~ B ; BEGIN FOR i=1 TO q DO B'i=Bi; //Queries only occupy previous size (1) //No set is proposed to be cached first FOR i=1 TO m DO D i=0; (2) AvailB = B - (~,,=i/~ + Y.,"=l/~); //The total free buffer number (3) REPEAT (4) FOR i=1 TO q DO //Compute effects of adding query buffer (5) (6) Effecti = max~l. ...,Avo,m ( EfflQ,, Bi, b) ); MoxB~ = the b with maximum cost saving; (7) FOR i=1 TO m DO //Estimate effects of caching derived sets (8) IF (D',~ 0 or Si>AvailB) THEN Effectq+i = 0 (9) (10) ELSE Effectq+i = E T X Profit(M,); //Then execute the most effective cache allocation IF (some Mi has maximum effec0 THEN (11) {D', = S, ; AvailB = AvailB - S~ } (12) ELSE (some Q, has maximum effec0 (13) {B ,=B , + M a x B g AvailB = AvailB - MaxB, }; (14) UNTIL (AvialB==O or max'.maurn,effect==0); (15) R E T U R N (13 1, B z, ..., B q, D l, D 2..... D'm); (16) END m
'
The algorit. ~ takes the initial allocation scheme as it input and produces a new scheme, (B 1, B 2, ..., B'q, D'1, D'2..... D'm). It satisfies B',>_B,, since that once the buffer pages have been provided to Q, it will have been using them, so the pages can not be deprived from it before it terminates. But D'~ can change from S~, to 0, which means that the cached DS, is to be evicted out according to the new scheme. If D', changes from 0 to S,, then the new scheme advises to cache DS,. The algorithm is written in pseudo code. It's a repetitive procedure. Each loop finds the most beneficial addition, i.e., cache some derived set or increase a query's buffer size. This repetitive procedure terminates if no buffer can be allocated, or no cost saving can be gained. Its major cost is that of cost saving computation, especially judging the most effective buffer size of Qj in line (6), which attempts every possible buffer number. An improvement is to adopt a binary search, which leads to sub-optimal answers. In fact, with the hit rate concavity assumption in 1, we can simplify this search. It need only compute the maximum effect of one-page buffer addition to all query instances. Above greedy algorithm is the kernel of cache allocation. Other algorithms rely on it, e.g., CA for allocating cache to new queries, which will be discussed below.
85
Another allocation algorithm EA in our design is used to extend the query's buffer space for more efficient processing. It is useful since some instances will terminate and release their occupied cache. It decides a requesting query instance's extended cache size by also calling GA to obtain the current best scheme.
Cache Allocation for New Queries On the basis of the greedy algorithm, the rather simple algorithm CA is described below. It re-computes the allocation scheme when a new query requires cache. The output of CA is the query instance's allocated cache size, and other query instances' cache sizes are not adjusted. However, all D'/s can be different to their previous values. It indicates that, some derived sets should be evicted out of the cache to support more efficient query processing, including that of the new query. Algorithm CA: The algorithm allocating cache to new queries Input: CAS=(BI, B2..... Bq, DI, 1)2..... Din), new query Qq+1, its minimum cache requirement Brainand page fault .function,PFr Output: CAS=(BI, B2, ..., Bq, B q+l, D I, D 2..... D m) and return Bq+t if allocate successfully; otherwise no update to CAS and report failure; BEGIN IF Bmm > B - Y.1~tBi THEN Report no available buffer and terminate; (1)
(2) (3) (4)
(B'I,~'2.....B'q,B'q+l,D'I.....D'ra) = GA(,(BI, B2 .....Bq, Brain,0 .....0) ); C A S = (Bb B2 .....Bq, B q+l, D'b D 2.....Dm); RETURN (B'q+t);
END If current available cache can afford the minimum requirement of query Qq+~,algorithm CA will call GA. The input of GA is the minimum fixed cache sizes occupied by the query instances, i.e., Bt for Qi (i=l., "", q) and Bminfor Qq+l. The execution of GA outputs a new scheme, where each B i or D, can differ to previous Bi or D~. After that CA generates a new scheme: B q+t buffer pages are allocated to the new query; other query instances maintain previous cache sizes; and the updated D'~ are included. This new scheme reflects the needs of current workload. Consider why the new scheme doesn't include the new cache sizes for other query instances. First, once a query instance has occupied some cache and is processing, it usually can not use additional buffer since its access method only utilizes the original cache. Thus it is wasteful to allocate the available buffer pages to it because they can be dedicated to possible new queries. Furthermore, we should note, these buffer pages, even if not allocated, are still in use as the free portion of the global cache pool.
4.2 Cache Admission and Replacement
The buffer scheduler is an internal algorithm of the global cache manager. When too many queries are in the system, they may contest the limited buffer pages. Some might
86 be blocked if current available buffer pages are not enough to meet their minimum requirements. When a query completes, its buffer page number will be returned to the cache manager. At this point the scheduler begins to work. It selects one or more buffer-waiting instances according to some criteria, and wakes up them. It also decides adequate cache sizes for these queries. For this, it adds new elements into the cache allocation scheme and calls algorithm GA. In our method, replacement strategies are necessary at two levels. The first level, private cache replacement, means that the query instances can adopt efficient replacement strategies fitting their access patterns. The query optimizer and processing program, instead of our global cache manager determine this level, so it is not the main topic of this paper. Let us consider the other level, global cache replacement. It seems that this level is unimportant since the available cache is usually allotted to the query instances as their private buffer pages. Contrary, it has great effect on the performance since: 9 Not all buffer pages serve for the query instances. The global cache, excluding the allocated parts, can be a large portion of the buffer pool. Many pages cache DS~'s. Buffer sharing also causes the free portion larger than expected. 9 Different types of free buffer pages have unequal values of being kept in cache. Some are the query instances' wasted pages; others are used to keep the derived sets. They require adequate replacement strategies. A simple cache replacement strategy is LRU, which is a widely studied and adopted approach in operating system and database system fields. As pointed out in 15, this replacement algorithm performs inadequately in the presence of multiple workload classes, due to the limited reference time information it uses in the selection of replacement victims. Consequently, the LRU-K algorithm has been proposed, which considers the times of the last K>I references to each page. We can also adopt this method. These two methods have a side effect. They are page-oriented, or they swap out the buffer pages in individual pages. Hence, usually a derived set is partially swapped out of the cache though the allocation scheme proposes to cache it. Some derived sets might be wholly evicted out of memory. We simply allow this inconsistency between the scheme and the replacement algorithm's actual results. We can also adopt another strategy called CAS-prior. Contrary to the LRU-K's ignoring the allocation scheme, this strategy aims to keep the derived sets with higher benefits prior to other database pages and derived sets. According to this strategy, if in the scheme Dj=S~, then DS~ should still in cache if it resides currently. Replacement algorithm only considers other individual pages and set data. From these candidates, the replacement algorithm can use LRU-K strategy for selection. The CAS-prior strategy does not rule that the derived sets declared in the allocation scheme are exactly those cached. A derived set proposed in cache can be out of memory if it has not been referenced recently. On the other hand, other data can also be in cache. When a query terminates, its buffer pages will still cache its used data before they are evicted out. This query can also have referenced derived sets that are not suggested to reside in memory.
87
5
Experimental Study
5.1 Experimental Environment We made experiments on the top of a client-server database management system. The system ran on a SGI Indigo 2 workstation. We loaded a medium size database and derived data, designed a multi-class workload stream to verify the performance of the global cache manager. A database of about 140M bytes was created. It contains 1,000,000 records, and each record is 100 bytes in size. The database is organized using a B+-tree index. All data and index page size is 4K bytes, the same as a buffer page.
We also defmed 10 query templates and stored the data sets in the disks. These query templates have different sizes and reference rates. Table 1 gives their storage sizes, reference rates, and retrieval costs. It is designed that the smaller views have greater probabilities to be referenced, but their retrieval costs are proportional to the sizes. The total size of the derived sets is 4,000 pages. We assumed the costs in Table 1 contribute mainly to the I/O costs, since we designed that the derived sets were stored in the disks and would be retrieved by scanning them. Thus unlike 18, where cost saving is the main metric, we simplify the problem and the hit rate becomes a reasonable performance metric.
5.2 Workload Design We designed a multi-class workload on both database and derived data. Table 2 gives the frequencies in QPS (Queries Per Second) of all five kinds of queries. Two kinds of simple queries on the base database are: 9 Q,: Exact match query. It searches only one record through the B+-tree by matching a randomly generated key value. 9 Q2: Range query. It scans a database segment (its range is also randomly decided), accessing 100 data pages.
88 w e also designed other three kinds of queries on the derived data. These queries can be translated into operations to the sets. They stochastically pick some derived sets. The sets' probabilities of being selected are proportional to their reference rates in Table 1. These three types of queries are: 9
Q3: Simple scan query, which scans a randomly selected set.
9
Q4: Sort-based aggregate-query of a randomly selected set.
9 Qs: Sort-merge join of two randomly selected sets. We mixed these queries in a query stream, and ran many versions of this stream simultaneously (Note, these versions are not simple duplications but actually different streams due to the random number generator). To ensure good effects, we ran as many versions as possible, e.g., up to 30 or 50 if the available system resources permit. Each version's query frequency can be computed through dividing 14.042 QPS by the number of concurrent versions. Table 2. Query Frequencies
Queries QPS
Ol 10
~2. 0.2 .......
Q3 3
Q4 0.8
05 0.042
5.3 P e r f o r m a n c e Metric
We used the primary performance metric page hit rate (HR) during some processing period. It's defined as:
HR =
TotalPageHit TotalPageReference
(13)
We converted the hit rate of derived sets into the uniform hit rate of individual pages. The cache manager monitors I/O operations and page references. Finally, they are summed up to compute HR. To obtain more accurate results, we ran the query streams for a long period from 20 minutes to an hour. The parameters are sampled from hot system when the cache manager has become experienced. Three GCM-based approaches are compared with other two non-GCM-based approaches. The latter two are LRU and LRU-K (K=3) without cache allocation mechanism. All query instances fully share and compete for the global cache. On the other hand, three GCM-based approaches are (detailed description in section 4): 9 GCM-LRU, where the global cache manager uses page LRU replacement for the free portion of the buffer pool. 9 GCM-LRU-K, where the global cache manager uses page LRU-K replacement for the free portion. 9 GCM-CAS-prior, where the global cache manager uses CAS-prior replacement for the free portion.
89
0.6 0.5 0.4 0.3 0.2
.%
0.1 "-
0
I
2
,
t
4
,
I
8
. . . . . .
I.
. . . . . . . . . .
12
GCM-CAS-pdo l
.
.
.
.
.
.
:
16 Cache siz
(M bytes)
Figure 2. Hit rates of the algorithms 5.4 Experimental Results Figure 2 gives the experimental results. The hit rates of simple LRU are always the worst for any cache size, and they are well below the others. Although the LRU-K method has not adopted adequate cache allocation for multiple queries, we f'md that LRU-K strategy's hit rates are almost two times high as those of LRU. Obviously LRU-K benefits from more historical information than LRU. A larger K improves the estimation of the individual page reference rates, consequently leads to a higher HR. We should point out that a larger K could play a more significant role under multiclass workloads in which each class has different reference characteristics. GCM-LRU approach has acceptable hit rates. R implements cache allocation for multiple queries and enables flexible cache replacement. But sometimes it is worse than the LRU-K strategy. This proves again the high performance of LRU-K. Another phenomenon is, when the cache size is small, its hit rates are obviously lower than LRU-K's, but if the buffer pool is enlarged, GCM-LRU closes, even outperforms the latter. Compared with GCM-LRU, GCM-LRU-K has consistently higher hit rates for all cache sizes. We fred that it is obviously better than the simple LRU-K strategy. The principal reason is, it combines efficient cache allocation for multiple queries and the experienced LRU-K replacement. Each query instance has its private cache replacement adaptable to its reference pattern. However, the most successful one is GCM-CAS-prior. It is slightly better than GCM-LRU-K. In GCM-LRU or GCM-LRU-K, the derived sets are possible to be evicted out of the global cache wholly or partly ignoring the cache allocation algorithms' propositions. This phenomenon becomes more possible when some costly derived sets are not repetitively referenced. Contrary, GCM-CAS-prior adopts effect
90 estimation and caches the most beneficial data, and the replacement algorithm retains the sets declared in current allocation scheme. In above experiments, we assumed that the execution overhead of query templates principally contributes to the retrieval I/O cost. In fact the computation of small sets can also be costly. If so, our GCM-CAS-prior approach will be more evidently superior to GCM-LRU-K and the others. We have not experimented to compare our method with that of 18 partly because we have not implemented their algorithms. Their performance analysis was also strong with respect to queries and data, but they had not considered buffer requirement of query instances. Actually when other operations ask for large amount of cache, the performance of caching derived sets will be affected significantly.
6
Conclusion
This paper investigates the problem of global cache management for multiple queries in data warehouses. A practical comparable profit model is developed to compare the potential benefits of caching retrieved sets and buffering for query operations. Algorithms are designed to gain efficient cache allocation schemes, which arbitrate the balance between above two needs. Together with different replacement strategies, these algorithms generate several approaches. Experimental results indicate that our methods have better performances than simple LRU and LRU-K strategy. Researches on the following topics should be further investigated: * Effects of warehouse modifications. In this paper we have not considered their possible effects on cache management in warehouses, though updates are not fiequent in such environments. * Load control problem. Our global cache manager provides some load control mechanism. Each query instance requires a minimum cache size, so concurrent query number is limited. But R is not enough yet. 9 Benchmark performance evaluation. We hope to make experiments based on TPCD or Set Query benchmarks.
References 1. Brown, K.P., Carey, M.L, Livny, M.: Goal-oriented buffer management revisited. In Proceedings ofACM SIGMOD Conference, 1996:353-364 2. Baralis,E., Paraboschi, S., Teniente,E.: Materialized view selection in a multidimensional database. In Proceedings of VLDB Conference, 1997:156-165 3. Chou, H., DeWitt, D.J.: An evaluation of buffer management strategies for relational database systems. In Proceedings of VLDB Conference, 1985:127-141 4. Chan, C., Ooi, B., Lu, J.: Extensible buffer management of indexes. In Proceedings of VLDB Conference, 1992:444-454
91 5.
Chen, C., Roussopoulos, N.: The implementation and performance evaluation of the ADMS query optimizer: Integrating query result caching and matching. In Proceedings of EDBT Conference, 1994:323-336 6. Faloutsos, C., Ng, R., Sellis, T.: Predictive load control for flexible buffer allocation. In Proceedings of VLDB Conference, 1991:265-274 7. Faloutsos, C., Ng, R., Sellis, T.: Flexible and adaptable buffer management techniques for database management systems. IEEE Trans. on Computers, 44(4), 1995:546-560 8. Gupta, A., Harinarayan, V., Quass, D.: Aggregate-query processing in data warehousing environments. In Proceedings of VLDB Conference, 1995:358-369 9. Gupta, A., Mumick, I.: Maintenance of materialized views: Problems, techniques, and applications. IEEE Data Engineering Bulletin, 18(2), 1995:3-18 10. Gupta, A.: Selection of views to materialize in a data warehouse. In Proceedings of ICDT Conference, 1997:98-112 11. Harinarayan, V., Rajaraman, A., Ullman, J.: Implementing data cubes efficiently. In Proceedings of SIGMOD Conference, 1996:205-216 12. Huyn, N.: Multiple-view self-maintenance in data warehousing environments. In Proceedings of VLDB Conference, 1997:26-35 13. Mohan, N.: DWMS: Data warehouse management system. In Proceedings of VLDB Conference, 1996:588 14. Ng, R., Faloutsos, C., Sellis, T.: Flexible buffer allocation based on marginal gains. In Proceedings of SIGMOD Conference, 1991:387-396 15. O'Neil, E., O'Neil, P,, Weikum, G.: The LRU-K page replacement algorithm for database disk buffering. In Proceedings of SIGMOD Conference, 1993: 297-306 16. Sellis, T.: Intelligent caching and indexing techniques for relational database systems. Information Systems, 13(2), 1988:175-185 17. Staudt, M., Jarke, M.: Incremental maintenance of externally materialized views, In Proceedings of VLDB Conference, 1996:75-86 18. Scheuermann, P., Shim, J., Vingralek, R.: WATCHMAN: A data warehouse intelligent cache manager. In Proceedings of VLDB Conference, 1996:51-62 19. Widom, J.: Research problems in data warehousing. In: Proceedings of CIKM Conference, 1995.
A r c h i t e c t u r e a n d Q u a l i t y in D a t a W a r e h o u s e s 1 Matthias Jarke(~), Manfred A. Jeusfeld(z) Christoph Quix (~), Panos Vassiliadis (3) (1) RWTH Aachen, Germany, {jarke,quix} @informatik.rwth--aachen.de (2) Tilburg University, The Netherlands, [email protected] (3) National Technical University of Athens, Greece, [email protected]
Abstract. Most database researchers have studied data warehouses (DW) in
their role as buffers of materialized views, mediating between updateintensive OLTP systems and query-intensive decision support. This neglects the organizational role of data warehousing as a means of centralized information flow control. As a consequence, a large number of quality aspects relevant for data warehousing cannot be expressed with the current DW meta models. This paper makes two contributions towards solving these problems. Firstly, we enrich the meta data about DW architectures by explicit enterprise models. Secondly, many very different mathematical techniques for measuring or optimizing certain aspects of DW quality are being developed. We adapt the Goal-Question-Metric approach from software quality management to a meta data management environment in order to link these special techniques to a generic conceptual framework of DW quality. Initial feedback from ongoing experiments with a partial implementation of the resulting meta data structure in three industrial case studies provides a partial validation of the approach.
1
Introduction
Data warehouses provide large-scale caches of historic data. They sit between information sources gained externally or through online transaction processing systems (OLTP), and decision support or data mining queries following the vision of
i This research was partially supported by the European Commission in ESPRIT Long Term Research Project DWQ (Foundations of Data Warehouse Quality), by the General Secretariat of Research and Technology (Greece) under the PENED program; and by the Deutsche Forschungsgemeinschaft through Graduiertenkolleg "Informatik und Technik".
94 online analytic processing (OLAP). Three main arguments have been put forward in favor of this caching approach:
1. Performance and safety considerations: The concurrency control methods of most DBMSs do not react well to a mix of short update transactions (as in OLTP) and OLAP queries that typically search a large portion of the database. Moreover, the OLTP systems are often critical for the operation of the organization and must not be under danger of corruption of other applications.
2. Logical interpretability problems: Inspired by the success of spreadsheet techniques, OLAP users tend to think in terms of highly structured multidimensional data models, whereas information sources offer at best relational, often just semi-structured data models.
3. Temporal and granularity mismatch: OLTP systems focus on current operational support in great detail, whereas OLAP often considers historical developments at a somewhat less detailed granularity. Thus, quality considerations have accompanied data warehouse research from the beginning. A large body of literature has evolved over the past few years in addressing the problems introduced by the DW approach, such as the trade-off between freshness of DW data and disturbance of OLTP work during data extraction; the minimization of data transfer through incremental view maintenance; and a theory of computation with multi-dimensional data models. However, the heavy use of highly qualified consultants in data warehouse applications indicates that we are far from a systematic understanding and usage of the interplay between quality factors and design options in data warehousing. The goal of the European DWQ project JV97 is to address these issues by developing, prototyping and evaluating comprehensive Foundations for Data Warehouse Quality, delivered through enriched meta data management facilities in which specific analysis and optimization techniques are embedded. This paper develops the DWQ architecture and quality management framework and describes first steps towards its implementation and validation. The main contributions include an extension of the standard DW architecture used in the literature by enterprise modeling aspects, and a strategy for embedding specialpurpose mathematical reasoning tools in the architecture which will enable a computationally tractable yet rich quality analysis or quality-driven design process. Interaction with DW tool vendors, DW application developers and administrators has shown that the standard framework used in the DW literature is insufficient to capture in particular the business role of data warehousing. A DW is a major investment made
95 to satisfy some business goal of the enterprise; quality model and DW design should reflect this business goal as well as its subsequent evolution over time. In section 2, we discuss this problem in detail; our new architectural framework separates (and links) explicitly the concerns of conceptual enterprise perspectives, logical data modeling (the main emphasis of DW research to date), and physical information flow (the main concern of commercial DW products to date). In section 3, we first build on literature frameworks for data and software quality to come up with a suitable set of DW quality dimensions, as perceived by different groups of stakeholders. We then adapt a variant of the so-called Goal-QuestionMetric approach used in software quality management. Through materialized quality views, we link conceptual quality goals to specific analysis techniques developed in DW research and practice, and enable trade-offs between heterogeneous quality goals. Initial experiences with a prototypical implementation of the resulting meta database using the ConceptBase deductive object manager have been gained in cooperation with industrial case studies. Section 4 relates our approach to other work in data warehousing, data and software quality, while section 5 provides a summary and conclusions.
2
T h e Architecture o f a D a t a W a r e h o u s e
Physically, a data warehouse system consists of databases (source databases, materialized views in the distributed data warehouse), data transport agents that ship data from one database to another and a data warehouse repository which stores all kinds of meta data about the system. The content of the repository determines to a large extent how the data warehouse system can be used and evolved. The main goal of our approach is therefore to define a meta database schema which can capture and link all relevant aspects of DW architecture and quality. We shall tackle this very difficult undertaking in several steps.
2.1
Three Perspectives of Data Warehouse Meta Data
Almost all current research and practice understand a data warehouse architecture as a stepwise information flow from information sources through materialized views towards analyst clients, as shown in figure 2.1. Our key observation is that this architecture covers only partially the tasks faced in data warehousing and is therefore unable to even express, let alone support, a large number of important quality problems and management strategies.
96 Clients /
/
,/
/
\\
jJ~
~_~__/ .._________~ Data Warehouse
Adminstration ~ent
Wrappers/",, Loaders \"\""', i Sources Figure 2.1: Current Understanding of a Data Warehouse
As a consequence, we propose a separation of three perspectives as shown in figure 2.2: a conceptual enterprise perspective, a logical data modeling perspective, and a physical data flow perspective. The main argument we wish to make is the need for a conceptual enterprise perspective. To explain, consider the left two columns of figure 2.2. Suppose an analyst wants to know something about the business -- the question mark in the figure. She does not have the time to observe the business directly but must rely on existing information gained by operational departments, and documented as a side effect of OLTP systems. This way of information gathering implies already a bias which needs to be compensated when selecting OLTP data for uploading and cleaning into a DW where it is then further pre-processed and aggregated in data marts for certain analysis tasks. Considering the long path the data has taken, it is obvious that also the last step, the formulation of conceptually adequate queries and the conceptually adequate interpretation of the answers presents a major problem to the analyst. The traditional DW literature only covers two of the five steps in figure 2.2. Thus, it has no answers to typical practitioner questions such as "how come my operational departments put so much money in their data quality, and still the quality of my DW is terrible?" (answer: the enterprise views of the operational departments are not easily compatible with each other or with the analysts view), or "what is the effort required
97 to analyze problem X for which the DW currently offers no information?" (could simply be a problem of wrong aggregation in the materialized views, could require access to not-yet-integrated OLTP sources, or even involve setting up new OLTP sensors in the organization). An adequate answer to such questions requires an explicit model of the conceptual relationships between an enterprise model, the information captured by OLTP departments, and the OLAP clients whose task is the decision analysis. We have argued that a DW is a major investment undertaken for a particular business purpose. We therefore do not just introduce the enterprise model as a minor part of the environment, but demand that all other models are defined as views on this enterprise model. Perhaps surprisingly, even information source schemas define views on the enterprise model -- not vice versa as suggested by figure 2.1 !
Figure 2.2 The Data Warehouse Meta Data Framework
The wrapping and aggregation transformations performed in the (traditionally discussed) logical perspective can thus be checked for interpretability, consistency or completeness with respect to the enterprise model -- provided an adequately powerful representation and reasoning mechanism is available. At the same time, the logical transformations need to be implemented safely and efficiently by physical storage and transportation -- the third perspective in our approach. It is clear that physical quality aspects require completely different modeling formalisms than the conceptual factors, typical techniques stemming from queuing theory and combinatorial optimization.
98 There is no single decidable formalism that could handle all of these aspects uniformly in a meta database. We have therefore decided to capture the architectural framework in a deductive object data model in a comprehensive but relatively shallow manner. Special-purpose reasoning mechanisms such as the ones mentioned above can be linked to the architectural framework as discussed in section 3, below.
2.2
A Notation for Data Warehouse Architecture
We use the meta database to store an abstract representation of data warehouse applications in terms of the three-perspective scheme. The architecture and quality models are represented in Telos [MBJK90], a metadata modeling language. Its implementation in the ConceptBase system [JGJ+95] provides query facilities, and definition of constraints and deductive rules. Telos is well suited because it allows to formalize specialized modeling notations by means of meta classes. Preloaded with these metaclasses, the ConceptBase system serves as the meta database for qualityoriented data warehouses. A condensed graphical overview of the architecture notation is given in Figure 2.3. Bold arrows denote specialization links. The most general meta class is DW_Object. It subsumes objects at any perspective (conceptual, logical, or physical) and at any level (source, data warehouse, or client).
Figure 2.3: Overview of the Architecture Notation
Within each perspective, we distinguish between the modules it offers (e.g. client model) and the kinds of information found within these modules (e.g. concepts and their subsumption relationships). The horizontal links hasSchema and isViewOn establish the way how the horizontal links in Figure 2.2 are interpreted: the types of a
99 schema (i.e., relational or multidimensional structures) are defined as logical views on the concepts in the conceptual perspectives. On the other hand, the components of the physical perspective get a schema from the logical perspective as their schema. Each
object
can
have an associated set of materialized views called QualityMeasurements. These materialized views (which can also be specialized to the different perspectives -- not shown in the figure) constitute the bridge to the quality model discussed in section 3. The horizontal levels of the objects are coded by the three subclasses attached to
Model, Schema, and DataStore. We found this notation adequate to represent physical data warehouse architectures of commercial applications, such as the SourcePoint tool marketed by Software AG SAG96 or the DW architecture underlying a data mining project at Swiss Life SKR97. The logical perspective currently supports relational schema definitions whereas the conceptual perspective supports the family of extended entity-relationship and similar semantic data modeling languages. Note that all objects in Figure 2.3 are meta classes: actual conceptual models, logical schemas, and data warehouse components are represented as instances of them in the meta database. In the following subsections, we elaborate on the purpose of representing each of the three perspectives.
2.3
Conceptual Perspective
The conceptual perspective describes the business models underlying the information systems of an enterprise. The central role is played by the enterprise model, which gives an integrative overview of the conceptual objects of an enterprise. The models of the client and source information systems are views on the enterprise model, i.e. their contents are described in terms of the enterprise model. One goal of the conceptual perspective is to have a model of the information independent from physical organization of the data, so that relationships between concepts can be analyzed by intelligent tools, e.g. to simplify the integration of the information sources. On the client side, the interests of user groups can also be described as views on the enterprise model. In the implementation of the conceptual perspective in the meta database, the central class is Model. A model is related to a source, a client or the relevant section of the enterprise, and it represents the concepts which are available in the corresponding source, client or enterprise. The classes ClientModel, SourceModel and EnterpriseModel are needed, to distinguish the models of several sources, clients and the enterprise itself. A model consists of Concepts, each representing a concept of the
100
real world, i.e. the business world. If the user provides some information about the relationship between concepts in a formal language like description logics, a reasoner can check for subsumption of concepts CDL97. The results of the reasoning process are stored in the model as attribute isSubsumedBy of the corresponding concepts. Essentially, the repository can serve as a cache for reasoning results. Any tool can ask the repository for containment of concepts. If the result has already been computed, it can directly be answered by the repository. Otherwise, a reasoner is invoked by the repository to compute the result.
2.4
Logical Perspective
The logical perspective conceives a data warehouse from the view point of the actual data models involved, i.e. the data model of the logical schema is given by the corresponding physical component, which implements the logical schema. The central point in the logical perspective is Schema. As a model consists of concepts a schema consists of Types. We have implemented the relational model as an example for a logical data model; other data models such as the multi-dimensional or the objectoriented data model are also being integrated in this framework. Like in the conceptual perspective, we distinguish in the logical perspective between ClientSchema, DWSchema and SourceSchema for the schemata of clients, the data warehouse and the sources. For each client or source model, there is one corresponding schema. This restriction is guaranteed by a constraint in the architecture model. The link to the conceptual model is implemented by the relationship between concepts and types: each type is expressed as a view on concepts.
2.5
Physical Perspective
Data warehouse industry has mostly explored the physical perspective, so that many aspects in the physical perspective are taken from the analysis of commercial data warehouse solutions such as Software AG's SourcePoint tool SAG96, the data warehouse system of RedBrick RedB97, Informix's MetaCubeInfo97, Essbase of Arbor Software Arbo96 or the product suite of MicroStrategy MStr97. We have observed that the basic physical components in a data warehouse architecture are agents and data stores. Agents are programs that control other components or transport data from one physical location to another. Data stores are databases which store the data that is delivered by other components.
101
The basic class in the physical perspective is DW_Component. A data warehouse component may be composed out of other components. This fact is expressed by the attribute hasPart. Furthermore, a component deliversTo another component a Type, which is part of the logical perspective. Another link to the logical model is the attribute hasSchema of D W_Component. Note that a component may have a schema, i.e. a set of several types, but it can only deliver a type to another component. This is due to the observation that agents usually transport only "one tuple at a time" of a source relation rather than a complex object. One type of component in a data warehousing environment is an Agent. There are two types of agents: ControlAgent which controls other components and agents, e.g. it
notifies another agent to start the update process, and TransportationAgent which transports data from one component to another component. An Agent may also notify other agents about errors or termination of its process. Another type of component is a DataStore. It physically stores the data which is described by models and schemata in the conceptual and logical perspective. As in the other perspectives, we distinguish between ClientDataStore, DW_DataStore and SourceDataStore for data stores of clients, the data warehouse and the sources.
3
Managing Data Warehouse Quality
In this section, we discuss how to extend the DW architecture model by explicit quality models and their support. There are two basic issues to be resolved. On the one hand, quality is a subjective phenomenon so we must organize quality goals according to the stakeholder groups that pursue these goals. On the other hand, quality goals are highly diverse in nature. They can be neither assessed nor achieved directly but require complex measurement, prediction, and design techniques, often in the form of an interactive process. The overall problem of introducing quality models in meta data is therefore to achieve breadth of coverage without giving up the detailed knowledge available for certain criteria. Only if this combination is achieved, systematic quality management becomes possible.
3.1
Stakeholdersin Data Warehouse Quality
There exist different roles of users in a data warehouse environment. The Decision Maker usually employs an OLAP query tool to get answers interesting to him. A decision maker is usually concerned with the quality of the stored data, their timeliness and the ease of querying them through the OLAP tools. The Data
102
Warehouse Administrator needs facilities like error reporting, metadata accessibility and knowledge of the timeliness of the data, in order to detect changes and reasons for them, or problems in the stored information. The Data Warehouse Designer needs to measure the quality of the schemata of the data warehouse environment (both existing or newly produced) and the quality of the metadata as well. Furthermore, he needs software evaluation standards to test the software packages he considers purchasing. The Programmers of Data Warehouse Components can make good use of software implementation standards in order to accomplish and evaluate their work. Metadata reporting can also facilitate their job, since they can avoid mistakes related to schema information. Based on this analysis, we can safely argue that different roles imply a different collection of quality dimensions, which a quality model should be able to address in a consistent and meaningful way. In the following, we summarize the quality dimensions of three stakeholders, the data warehouse administrator, the programmer, and the decision maker. A more detailed presentation can be found in DWQ97b. Design and Administration Quality. The design and administration quality can be analyzed into more detailed dimensions, as depicted in Figure 3.1. The schema quality refers to the ability of a schema or model to represent adequately and efficiently the information. The correctness dimension is concerned with the proper comprehension of the entities of the real world, the schemata of the sources (models) and the user needs. The completeness dimension is concerned with the preservation of all the crucial knowledge in the data warehouse schema (model). The minimality dimension describes the degree up to which undesired redundancy is avoided during the source integration process. The traceability dimension is concerned with the fact that all kinds of requirements of users, designers, administrators and managers should be traceable to the data warehouse schema. The interpretability dimension ensures that all components of the data warehouse are well described, so as to be administered easily. The metadata evolution dimension is concerned with the way the schema evolves during the data warehouse operation. Software Implementation Quality. Software implementation and/or evaluation is not a task with specific data warehouse characteristics. We are not actually going to propose a new model for this task, but adopt the ISO 9126 standard ISO91. The quality dimensions of ISO 9126 are Functionality (Suitability, Accuracy, Interoperability, Compliance, Security), Reliability (Maturity, Fault tolerance, Recoverability), Usability (Understandability, Learnability, Operability), Software Efficiency (Time behavior, Resource Behavior), Maintainability (Analyzability, Changeability, Stability, Testability), Portability (Adaptability, Installability, Conformance, Replaceability).
103
Figure 3.1 Design and administration quality dimensions Data Usage Quality. Since databases and -in our case- data warehouses are built in order to be queried, the most basic process of the warehouse is the usage and querying of its data. Figure 3.2 shows the hierarchy of quality dimensions related to data usage.
Figure 3.2 Data usage quality dimensions
accessibility dimension is related to the possibility of accessing the data for security dimension describes the authorization policy and the privileges each user has for the querying of the data. System availability describes the percentage The
querying. The
of time the source or data warehouse system is available (i.e. the system is up and no backups take place, etc.). The transactional availability dimension, as already mentioned, describes the percentage of time the information in the warehouse or the source is available due to the absence of update processes which write-lock the data.
104
The usefulness dimension describes the temporal characteristics (timeliness) of the data as well as the responsiveness of the system. The responsiveness is concerned with the interaction of a process with the user (e.g. a query tool which is self reporting on the time a query might take to be answered). The currency dimension describes when the information was entered in the sources or/and the data warehouse. The volatility dimension describes the time period for which the information is valid in the real world. The interpretability dimension, as already mentioned, describes the extent to which the data warehouse is modeled efficiently in the information repository. The better the explanation is, the easier the queries can be posed.
3.2
From Architecture to Quality
We now turn to the formal handling and repository-based management of DW quality goals such as the ones described in the previous section. A first formalization could be based on a qualitative analysis of relationships between the quality factors themselves, e.g. positive or negative goal-subgoal relationships or goal-means relationships. The stakeholders could then enter their subjective evaluation of individual goals as well as possible weightings of goals and be supported in identifying good trade-offs. The entered as well as computed evaluations could be used as quality measurements in the architecture model of figure 2.3, thus enabling a very simple integration of architecture and quality model. Such an approach is widely used in industrial engineering under the label of Quality Function Deployment, using a special kind of matrix representation called the House of Quality Akao90. Formal reasoning in such a structure has been investigated in works about the handling of non-functional requirements in software engineering, e.g. MCN92. Visual tools have shown a potential for negotiation support under multiple quality criteria GJJ97. However, while this simple approach certainly has a useful role in cross-criteria decision making, using it alone would throw away the richness of work created by research in measuring, predicting, or optimizing individual DW quality factors. In the DWQ project, such methods are systematically adopted or newly developed for all quality factors found important in the literature or our own empirical work. To give an impression of the richness of techniques to be considered, we use a single quality factor -- responsiveness in the sense of good query performance -- for which the DWQ project has studied three different approaches, one each from the conceptual, logical, and physical perspective.
105
We start with the logical perspective TS97.
Here, the quality indicator associated
with responsiveness is taken to be a weighted average of query and update "costs" for a given query mix and given information sources. A combinatorial optimization technique is then proposed that selects a collection of materialized views as to minimize the total costs. This can be considered a very simple case of the above Quality Function Deployment approach, but with the advantage of automated design of a solution. If we include the physical perspective, the definition of query and update "costs" becomes an issue in itself: what do we mean by costs -- response time, throughput, or a combination of both (e.g. minimize query response time and maximize update throughput)? what actually produces these costs -- is database access or the network traffic the bottleneck? A comprehensive queuing model NJ97 enables the prediction of such detailed metrics from which the designer can choose the right ones as quality measurements for his design process. In addition, completely new design options come into play : instead of materializing more views to improve query response time (at the cost of disturbing the OLTP systems longer at update time), the designer could buy a faster client PC or DBMS, or use an ISDN link rather than using slow modems. Yet other options come into play, if a rich logic is available for the conceptual perspective. The description logic DWQ uses for formalizing the conceptual perspective CDL97, allows to state that, e.g., information about all instances of one concept in the enterprise model is maintained in a particular information source, i.e. the source is complete with respect to the domain. This enables the DW designer to drop the materialization of all views on other sources, thus reducing the update effort semantically without any loss in completeness of the answers. It is clear that there can be no decidable formal framework that even comes close to covering all of these aspects in a uniform language. When designing the meta database extensions for quality management, we therefore had to look for another solution that still maintains the overall picture offered by the shallow quality management techniques discussed at the beginning of this section but is at the same time open for the embedding of specialized techniques. Our solution to this problem builds on the widely used Goal-Question-Metric (GQM) approach to software quality management OB92. The idea of GQM is that quality goals can usually not be assessed directly, but their meaning is circumscribed by questions that need to be answered when evaluating the quality. Such questions again can usually not be answered directly but rely on metrics applied to either the product or process in question; techniques such as statistical process control charts are then applied to derive the answer of a question from the measurements.
106
Our repository solution uses a similar approach to bridge the gap between quality goal hierarchies on the one hand, and very detailed metrics and reasoning techniques on the other. The bridge is defined through the idea of quality measurements as materialized views over the data warehouse which we already introduced in figure 2.3, and through generic queries over these quality measurements. This imp!ementation strategy provides more technical support than usual GQM implementations. It is enabled through the powerful parameterized query class mechanism offered by the ConceptBase system.
Figure 3.3: A notation for Data Warehouse Quality
The purpose of a quality goal is usually to improve some quality values of the DW or to achieve a certain quality value. Quality goals are associated with types of queries defined over quality measurements. These queries will support the evaluation of a specific quality goal when parameterized with a given (part of a) DW meta database. Such a query usually compares the analysis goal to a certain expected interval in order to assess the level of quality achieved. Furthermore, goals are established by stakeholders, who may have several subjective quality preferences. As a consequence, the quality measurement must contain information about both expected and actual values. Both could be entered into the meta database manually, or computed inductively by a given metric through a specific reasoning mechanism. For example, for a given physical design and some basic measurements of component and network speeds, the queuing model in [NJ97] computes the quality values for response time and throughput, and it could indicate if network or database access is the bottleneck in the given setting. This could then be combined with conceptual or logical quality measurements at the level of optimizing the underlying quality goal.
107
The interplay o f goals, queries, and metrics with the basic concepts of the architecture model is shown in the Telos meta model of figure 3.3. While the development and integration of numerous specific metrics is the goal of ongoing work in the DWQ project, our current implementation just covers the upper levels of the picture, such that only manual entry of quality measurements is supported. A number o.f quality queries have been implemented, focusing on some that turned out to be relevant when validating the architecture against three case studies: creating a model of Software AG's SourcePoint DW loading environment, modeling data quality problems hindering the application of data mining techniques in Swiss Life, and conceptually re-constructing some design decisions underlying the administrative data warehouses of the City of Cologne, Germany DWQ97a, DWQ97b. Quality queries access information recorded in quality measurements. A quality measurement stores the following information about data warehouse components: 1. an interval of expected quality measures 2. the current quality measure 3. the metric used to compute a measure 4. dependencies to other quality measurements The dependencies between quality measurements can be used to trace quality measurements outside the expected interval to their causes. The following two queries exemplify how quality measurements classify data warehouse components and how the backtracing of quality problems can be done by queries to the meta database: QueryClass BadQualityMeasurement isA QualityMeasurement with constraint c: $ not (this.expected contains this.current) $ end GenericQueryClass CauseOfBadQuality isA DW_Object with parameter badObject : DW_Object constraint c: $ exists ql,q2/QualityMeasurement (badObject measuredBy ql) and (ql in BadQualityMeasurement) and (ql dependsOn q2) and (q2 in BadQualityMeasurement) and ((this measuredBy q2) or (exists o/DW_Object (o measuredBy q2) and (this in CauseOfBadQualityo/badObject))) end
$
108
4
Related Work
Our approach extends and merges results from data warehouse research and data/software quality research. Starting with the data warehouse literature, the well-known projects have focused almost exclusively on what we call the logical and physical perspectives of DW architecture. While the majority of early projects have focused on source integration aspects, the recent effort has shifted towards the efficient computation and recomputation of multi-dimensional views. The business perspective is considered at best indirectly in these projects. The Information Manifold (IM) developed at AT&T is the only one that employs a rich domain model for information gathering from disparate sources such as databases, SGML documents, unstructured files LSK95, KLSS95, LRO96 in a manner similar to our approach. TSIMMIS (The Stanford-IBM Manager of Multiple Information Sources) is a project with the goal of providing tools for the integrated access to multiple and diverse information sources and repositories CGMH+94, U1197. Each information source is equipped with a wrapper that encapsulates the source, converting the underlying data objects to a common data model - called Object Exchange Model (OEM). On top of wrappers, mediators Wie92 can be conceptually seen as views of data found in one or more sources which are suitably integrated and processed. Similarly, but with slightly different implementation strategies, the Squirrel Project HZ96, ZHK96 provides a framework for data integration based on the notion of integration mediator. Integration mediators are active modules that support incrementally maintained integrated views over multiple databases. Moreover, data quality is considered by defining formal properties of consistency and freshness for integrated views. The WHIPS (WareHouse Information Prototype at Stanford) system HGMW+95, WGL+96 has the goal of developing algorithms for the collection, integration and maintenance of information from heterogeneous and autonomous sources. The WHIPS architecture consists of a set of independent modules implemented as CORBA objects. The central component of the system is the integrator, to which all other modules report. Turning to data quality analysis, Wang et al. WSF95 present a framework based on the ISO 9000 standard. They review a significant part of the literature on data quality, yet only the research and development aspects of data quality seem to be relevant to the cause of data warehouse quality design. In WRK95, an attribute-based model is presented that can be used to incorporate quality aspects of data products. The basis of
109
this approach is the assumption that the quality design of an information system can be incorporated in the overall design of the system. The model proposes the extension of the relational model as well as the annotation of the results of a query with the appropriate quality indicators. Further work on data quality can be found in BT89, BWPT93, Jans88, LU90, Hall78, Krie.79, and AA87. Variants of the Goal-Question-Metric (GQM) approach are widely adopted in software quality management OB92. A structured overview of the issues and strategies, embedded in a repository framework, can be found in JP92. Several goal hierarchies of quality factors have been proposed, including the GE Model MRW78 and Boeh89. ISO 9126 ISO91 suggests six basic factors which are further refined to an overall 21 quality factors. In HR96 a comparative presentation of these three models is offered and the SATC software quality model is proposed, along with metrics for all their software quality dimensions.
5
Discussion and Conclusions
The goal of our work is to enrich meta data management in data warehouses such that it can serve as a meaningful basis for systematic quality analysis and quality-driven design. To reach this goal, we had to overcome two limitations of current data warehouse research. Firstly, the basic architecture in which data warehouses are typically described turned out to be too weak to allow a meaningful quality assessment : as quality is usually detected only by its absence, quality-oriented meta data management requires that we address the full sequence of steps from the capture of enterprise reality in operational departments to the interpretation of DW information by the client analyst. This in turn implied the introduction of an explicit enterprise model as a central feature in the architecture. To forestall possible criticism that full enterprise modeling has proven a risky and expensive effort, we point out that our approach to enterprise model formation (including the formal language used in CDL97) is fully incremental such that it is perfectly feasible to construct the enterprise model step by step, e.g. as a side effect of source integration or of other business process analysis efforts. The second major problem is the enormous richness in quality factors, each associated with its own wealth of measurement and design techniques. Our quest for an open quality management environment that can accommodate existing or new such techniques led us to an adaptation and repository integration of the Goal-Question Metric approach where parameterized queries and materialized quality views serve as the missing link between specialized techniques and the general quality framework.
110
The power of the repository modeling language determines the boundary between precise but narrow metrics and comprehensive but shallow global repository. The deductive object base formalism of the Telos language provides a fairly sophisticated level of global quality analysis in our prototype implementation but is still fully adaptable and general; once the quality framework has sufficiently stabilized, a procedurally object-oriented approach could do even more, by encoding some metrics directly as methods, of course at the expense of flexibility. Conversely, a simple relational meta database could take up some of the present models with less semantics than offered in the ConceptBase system, but with the same flexibility. As of now, both the framework and its implementation can only be considered partially validated. One strain of current work therefore continues the validation against several major case studies, in order to set priorities among the quality criteria to be explicated in specific metrics and analysis techniques. A second overlapping strain concerns the development of these techniques themselves, and their linkage into the overall framework through suitable quality measurements and extensions to global design and optimization techniques. Especially when progressing from the definition of metrics and prediction techniques to actual design methods, it is expected that these will not be representable as closed algorithms but must take the form of interactive work processes defined over the DW architecture. As an example, feedback from at least two case studies suggests that, in practice, the widely studied strategy of incremental view maintenance in the logical sense is far less often problematic than the time management at the physical and conceptual level, associated with the question when to refresh DW views such that data are sufficiently fresh for analysis, but neither analysts nor OLTP applications are unduly disturbed in their work due to locks on their data. Our research therefore now focuses on extending the conceptual level by suitable (simple) temporal representation and reasoning mechanisms for representing freshness requirements, complemented by an array of design and implementation methods to accomplish these requirements and the definition of processes at the global level to use these methods in a goal-oriented manner to fulfill the requirements. While such extensions will certainly refine and in parts revise the approach reported here, the experiences gained so far indicate that it is a promising way towards more systematic and computer-supported quality management in data warehouse design and operation. Acknowledgements. The authors would like to thanks their project partners in DWQ, especially Maurizio Lenzerini, Mokrane Bouzeghoub and Enrico Franconi, for fruitful discussions of the architecture and quality model.
111
6
References
AA87 Akao90 Arbo96 Boeh89 BT89 BWPT93
CDL971
CGMH+94
DWQ97a
DWQ97b
GJJ97
Hall78 HGMW+95
N. Agmon, N.Ahituv, Assessing data reliability in an information system, J. Management Information Systems 4, 2 (1987) Akao, Y., ed., Qualio, Function Deployment, Productivity Press, Cambridge MA., 1990 Arbor Software Corporation. Arbor Essbase. http://www.arborsoft.com/essbase.html, 1996. Boehm, B., Software Risk Management, IEEE Computer Society Press, CA, 1989. D.P. Ballou, K.G. Tayi, Methodology for allocating resources for data quality enhancement, Comm. ACM, 32, 3 (1989) D.P. Ballou, R.Y. Wang, H.L. Pazer, K.G. Tayi, Modeling Data Manufacturing Systems To Determine Data Product Quality, (No. TDQM-93-09) Cambridge Mass.: Total Data Quality Management Research Program, MIT Sloan School of Management, 1993 D. Calvanese, G. De Giacomo, M. Lenzerini. Conjunctive query containment in Description Logics with n-ary relations. International Workshop on Description Logics, Paris, 1997. S. Chawathe, H. Garcia-Molina, J. Hammer, K. Ireland, Y. Papakonstantinou, J. Ullman, and J. Widom. The TSIMMIS project: Integration of heterogeneous information sources. In Proc. oflPSI Conference, Tokyo (Japan), 1994. DWQ, Deliverable DI.1, Data Warehouse Quality Requirements and Framework, NTUA, RWTH, INRIA, DFKI, Uniroma, IRST, DWQ TR DWQ - NTUA - 1001, 1997 DWQ, Deliverable D2.1, Data Warehouse Architecture and Quality Model, RWTH, NTUA, Uniroma, INRIA, DWQ TR DWQ - RWTH - 002, 1997 M. Gebhardt, M. Jarke, S. Jacobs, CoDecide -- a toolkit for negotiation support interfaces to multi-dimensional data. Proc. ACM-SIGMOD Conf. Management of Data, Tucson, Az, 1997. Halloran et al., Systems development quality control, MIS Quarterly, vol. 2, no.4, 1978 J. Hammer, H. Garcia-Molina, J. Widom, W. Labio, Y. Zhuge. The Stanford Data Warehousing Project. Data Eng., Special Issue Materialized Views on Data Warehousing, 18(2), 41-48. 1995.
112
HR96
HZ96
Info97
IS091
Jans88 JGJ+95
JP92
JV97
KLSS95
Krie79
LRO96
LSK95
LU90
L. Hyatt, L. Rosenberg, A Software Quality Model and Metrics for Identifying Project Risks and Assessing Software Quality, 8th Annual Software Technology Conference, Utah, April, 1996. R. Hull, G. Zhou. A Framework for supporting data integration using the materialized and virtual approaches. Proc. ACM SIGMOD Intl. Conf. Management of Data, 481 - 492, Montreal 1996. Informix, Inc.: The INFORMIX-MetaCube Product Suite. http://www.informix.com/informix/products/new plo/metabro/meta bro2.htm, 1997. ISO/IEC 9126, lnjbrmation technology -Software product evaluation- Quali~.' charactyeristics and guidelines for their use, International Organization for Standardization, http://www.iso.ch M. Janson, Data quality: The Achilles heel of end-user computing, Omega J. Management Science, 16, 5 (1988) M. Jarke, R. Gallersd6rfer, M.A. Jeusfeld, M. Staudt, S. Eherer: ConceptBase - a deductive objectbase for meta data management. In Journal of Intelligent Information Systems, 4, 2, 167-192, 1995. M.Jarke, K.Pohl. Information systems quality and quality information systems. In Kendall/Lyytinen/DeGross (eds.): Proc. IFIP 8.2 Working Conf. The Impact of Computer-Supported Technologies on Information Systems Development (Minneapolis 1992), North-Holland 1992, pp. 345-375. M. Jarke, Y. Vassiliou. Foundations of data warehouse quality -- a review of the DWQ project. Proc. 2nd Intl. Conf. Information Quality (IQ-97), Cambridge, Mass. 1997. T. Kirk, A.Y. Levy, Y. Sagiv, and D. Srivastava. The Information Manifold. Proc. AAA11995 Spring Symp. on Information Gathering from Heterogeneous, Distributed Environments, pp. 85-9 I, 1995. C. Kriebel, Evaluating the quality of inlbrmation system, Design and Implementation of Computer Based Information Systems, N. Szyperski/E.Grochla ,eds. Sijthoff and Noordhoff, 1979 A.Y. Levy, A. Rajaraman, and J.J. Ordille. Query answering algorithms for information agents. Proc. 13th Nat. Conf. on Artificial Intelligence (AAAI-96), pages 40-47, 1996. A.Y. Levy, D. Srivastava, and T. Kirk. Data model and query evaluation in global information systems. Journal of Intelligent Information Systems, 5:121-143, 1995. G.E. Liepins and V.R.R. Uppuluri, Accuracy and Relevance and the Quality of Data, A.S. Loebl, ed., vol. 112, Marcel Dekker, 1990
113
MBJK90
MCN92
MRW78 MStr97 INJ97
OB921
SAG96 SKR97
TS97 U11971
WGL+96
Wie92 WRK95 WSF95
ZHK96
J. Mylopoulos, A. Borgida, M. Jarke, M. Koubarakis: Telos - a language for representing knowledge about information systems.. In ACM Trans. Information Systems, 8, 4, 1990, pp. 325-362. J. Mylopoulos, L. Chung, B. Nixon. Representing and using nonfunctional requirements -- a process-oriented approach. 1EEE Trans. Software Eng. 18, 6 (1992). J.A. McCall, P.K. Richards, G.F. Waiters, Factors in software quality, Technical Report, Rome Air Development Center, 1978 MicroStrategy, Inc. MicroStrategy' s 4.0 Product Line. http://www.strategy.com/launch/4 0 arcl.htm, 1997. M. Nicola, M. Jarke. Integrating Replication and Communication in Performance Models of Distributed Databases. Technical Report, RWTH Aachen, AIB 97-10, 1997. M. Oivo, V. Basili: Representing software engineering models: the TAME goal-oriented approach. IEEE Trans. Software Eng. 18, 10 (1992). Software AG: SourcePoint White Paper. Software AG, Uhlandstr 12, 64297 Darmstadt, Germany, 1996. M. Staudt, J.U. Kietz, U. Reimer. ADLER: An Environment for Mining Insurance Data. Proc. 4th Workshop KRDB-97, Athens, 1997. D. Theodoratos, T. Sellis. Data Warehouse Configuration. Proc. 23th VLDB Conference, Athens, 1997. J.D. Ullman. Information integration using logical views. In Proc. 6th Int. Conf. on Database Theory (ICDT-97), Lecture Notes in Computer Science, pages 19-40. Springer-Verlag, 1997 J. L. Wiener, H. Gupta, W. J. Labio, Y. Zhuge, H. Garcia-Molina, J. Widom. A System Prototype for Warehouse View Maintenance. Proceedings ACM Workshop on Materialised Views: Techniques and Applications, Montreal, Canada, June 7, 1996, 26-33. G. Wiederhold. Mediators in the architecture of future information systems. IEEE Computer, pp. 38-49, March 1992. R.Y. Wang, M.P. Reddy, H.B. Kon, Towards quality data: an attribute-based approach, Decision Support Systems, 13(1995) R.Y. Wang, V.C. Storey, C.P. Firth, A framework for analysis of data quality research, IEEE Trans. Knowledge and Data Eng. 7, 4 (1995) G. Zhou, R. Hull, R. King. Generating Data Integration Mediators that Use Materialization. Journal of Intelligent Information Systems, 6(2), 199-221, 1996.
O M S / J a v a : Model Extensibility of O O D B M S for Advanced Application Domains Andreas Steiner, Adrian Kobler and Moira C. Norrie {steiner, kobler, norrie}@inf.ethz.ch Institute for Information Systems ETH Zurich, CH-8092 Zurich, Switzerland A b s t r a c t . We show how model extensibility of object-oriented data management systems can be achieved through the combination of a highlevel core object data model and an architecture designed with model extensibility in mind. The resulting system, OMS/Java, is both a general data management system and a framework for the development of advanced database application systems. All aspects of the core model constructs, query language and constraints - can easily be generalised to support, for example, the management of temporal, spatial and versioned data. Specifically, we show how the framework was used to extend the core system to a temporal object-oriented database management system.
1
Introduction
Extensibility has often been considered a purely architectural issue in database management systems (DBMS). In the 1980s, there was an increase in the various forms of DBMS that appeared - - many of which were tailored to specific application domains such as Geographical Information Systems or Computer Aided Design systems. It was recognised that no single DBMS could hope to satisfy the requirements of all such applications and the idea of an extensible DBMS emerged. Thereafter, a number of extensible systems appeared - including kernel e.g. CDKK85, customisable e.g. HCL+90, toolkit e.g. CDG+89 and generator e.g. MBH + systems. In these systems, the emphasis tends towards providing architectural support in terms of how to engineer new DBMS from existing frameworks and services such as storage management, transaction management, access methods and query optimisation and we therefore call this architecture extensibility. The idea evolved of a database engineer as someone whose task would be to construct an advanced DBMS based on such a system and, frequently, this was a significant task. A discussion of general requirements of such systems is given in GD94. In contrast, there have been proposals for various forms of model extensibility in which new types a n d / o r operators can be introduced in a DBMS to meet the needs of specific application domains, e.g. BLW88, SC94, Cha96. We aim for a higher-level of model extensibility in which all aspects of a data model structures, query language and constraints - can be generalised. This ensures upward compatibility in that the core model, languages and mode of working
116
remain essentially the same. For example, generalisation to a temporal system enables temporal semantics to be associated with all constructs, operations and constraints of the core model. Further, non-temporal queries will be unaffected and temporal queries have the same form as non-temporal queries - with only prefixes required to specify that a temporal semantics is required. Similarly, the DBMS could be generalised to handle spatial data, versioned data, multimedia data and any combination of these. OMS/Java is both a framework and an object data management system designed with such model extensibility in mind. The system is based on the OM model which, through a clear separation of model concepts, orthogonality of design and general modelling expressiveness has the properties required of the core model of such an extensible system. The development and architecture of OMS/Java exploited current software engineering principles such as application frameworks and design patterns to ensure that benefits of reusability and extensibility could be fully exploited. In this paper, we describe the OMS/Java architecture and system and how its extensibility was exploited to implement a temporal object-oriented database management system with minimal effort. We begin in Sect. 2 and 3 with a discussion of the general requirements for systems with true model extensibility. This includes a discussion of the requirements of an appropriate core model along with a description of our core model OM. Section 4 then presents the general OMS/Java architecture, followed with a description of the OMS/Java system. To demonstrate the process of model extensibility, we describe the development of the temporal OMS/Java system, TOMS/Java, in Sect. 5. We conclude with a summary and outline of future work in Sect. 6.
2 2.1
Architecture R e u s a b l e P a r t s of a D B M S
Modern DBMS have to support a variety of tasks and requirements EN94, HS95. A DBMS manages persistent data, meaning that data exists longer than tasks run by application programs. Data structures are defined to describe and store the data in a uniform way. Sophisticated access methods (for example, using indexing or hashing) provide for efficient data access. (Declarative) languages and operations need to be supported which allow the definition, modification and retrieval of data. The DBMS should also provide means and techniques to optimise data retrieval. Transaction management and concurrency control allow multiple users to access the same data simultaneously, avoiding any problems which might evolve. Additionally, a modern DBMS should support concepts to provide data consistency, recovery from system crashes and database security and authorisation. The implementation of an extensible data model as a DBMS should still support all of these tasks while allowing the user to extend the model and system with new functionality. Currently available DBMS support the features listed
117
above in one way or another. Relational DBMS support a single data structure the relation - and a non-extensible functionality. Object-oriented DBMS support extensibility of the data structures through the use of type constructors such as tuple, set, bag, list and so on and extensibility of the functionality through the possibility to store methods in the database. It is difficult to efficiently and effectively support application domains which are based on specialised data models - for example, the management of temporal or spatial data or versioning - using such systems. The limitations of the relational data model with respect to data structures and functionality make it necessary to either build the additional constructs needed into the application itself or implement a specialised DBMS. With respect to object-relational or object-oriented DBMS, it is possible to extend the system to support such advanced application domains. We have investigated such an approach in the area of temporal databases SN97b. The major drawbacks of this approach, however, are that the DBMS cannot use the special semantics - for example, the temporal semantics - for query optimisation, that the resulting retrieval and update operations turn out to be very complicated and error prone and that the specification of specialised integrity constraints usually may only be supported through the implementation of methods. Our idea of extensibility thus goes a step further. We would like to have a DBMS based on an extensible data model which allows the reuse of features such as persistence, query optimisation, efficient access methods, transaction management, crash recovery and authorisation, while supporting a generalised form of its data structures, algebra operations , query language and integrity constraints with special semantics. Such a system is upwards compatible in the sense that the basic data structures, operations and constraints can always be used, even in more advanced, generalised forms of the underlying data model, and that for each generalisation of the data model, the same concepts and manner of work can be used. The next section discusses in more detail the generalisation approach and introduces an architecture supporting this method of achieving more advanced data models. 2.2
The
Generalisation
Approach
A data model Mcan be considered to consist of three parts - the data structures DS, the operations for data retrieval and update 0P and the integrity constraints C, M = (DS, 0P, C). In SN97c, we have introduced a temporal data model which - in contrast to other temporal data models - enhances all three parts of a nontemporal data model to support the management of temporal data. Figure 1 shows the three parts of a data model. Most of the temporal data models extend non-temporal data models in that they add special timestamp attributes to store temporal data and add special operations to query temporal data. Other ones extend the data structures but supply a special temporal algebra. Our generalisation approach, however, considers all three parts of the data model by supporting temporal object identifiers, temporal algebra operations and temporal constraints. The resulting temporal object model is independent of any
118 type system and orthogonal in the sense that anything modelled as an object, including collections of objects, constraints and even types, can be timestamped.
DatSt aructures DatStaructures, GeneraUsatton of Algebraand Constraints
/ ~
Constraints
o ....
o ....
Alga
Algebraand
Constralnt~
Fig. 1. The three parts of a data model A DBMS should additionally provide for a language which can be used to specify the data structures and integrity constraints used for an application (the data definition language), update operations (the data manipulation language) and queries (the query language). This language thus consists of three sublanguages. In order to come up with a DBMS which allows the reuse of the general tasks of query optimisation, efficient access methods, transaction management, crash recovery and so on, but supplies generalised data structures, algebra operations and integrity constraints, the notion of an object identifier, the algebra operations, the integrity constraints and the language supported must be exchangeable, respectively, extendible. Such an architecture is depicted in Fig. 2.
!.=.
Core System
Fig. 2. Another form of extensibility in a DBMS
119
The core system supports the basic tasks of persistence, query optimisation, efficient access methods, transaction management, crash recovery, authorisation and so on. The object identifier, the algebra and constraints can be extended to support the needs of an advanced application domain, for example, temporal databases. In this case, the notion of an object identifier is generalised into a temporal object identifier which - besides a unique reference - contains a lifespan denoting when the object existed, for example with respect to the real world. To be able to query the temporal data stored in the form of temporal objects, the non-temporal algebra operations are overridden with algebra operations having the desired form of temporal semantics. Finally, the integrity constraints are replaced by temporal integrity constraints. The same approach can be used to support data models supporting the management of spatial data or versioning. For spatial data, we exchange, respectively, extend the notion of an object identifier with a spatial identifier, which means that the position of an object or the space that it covers is attached to the object identifier. Algebra operations and integrity constraints are replaced with operations and constraints which are able to deal with the new semantics. For versioning, the version number is attached to the object identifier and again, operations and integrity constraints which are able to deal with the versioned objects are added to the system.
3
Requirements
for Model
Extensibility
The generalisation approach, as introduced in the last section, obviously deals with objects and algebra operations and integrity constraints referring to these objects. Usually, it is argued that advanced application domains such as the management of temporal, spatial and versioned data can be supported using extensible DBMS which support abstract data types. Methods have to be written to support queries and constraints dealing with the special semantics of the application domain. Clearly, such approaches are based simply on the type system of the DBMS used. The disadvantage of these approaches are, for example, that the special semantics usually cannot be used for query optimisation, and that the specification of queries and integrity constraints in the core model and the advanced model are not similar. Our generalisation approach is based on the object level, which means, that we do not want to deal with instances of types and add additional attributes with special semantics. We want to deal with objects which inherently have special semantics, and a closer look at them reveals their property values. In other words, we advocate an approach which takes the special semantics of an advanced data model such as a temporal data model into account and does not treat temporal properties as just another, but somehow a bit more special, attribute value. Our idea of an extensible data model is that it should support a rich set of algebra operations and constraints which work on collections of objects. The following subsections discuss the requirements for model extensibility in more detail and introduce a data model which supports the generalisation approach.
120
3.1
Separation of Typing and Classification
Nor95 distinguishes between classifying entities into categories according to general concepts of the application domain and describing the representation of entities within the database in terms of types. For example, we can classify persons as being employees, students or tennis players. Each employee, student and tennis player is a person. Students can further be classified as studying either engineering or mathematics. This is depicted on the left hand side of Fig. 3. On the other hand, we represent persons as having a name and a birthdate, while employees additionally have an employment number and a salary, students have a student number and tennis players have a ranking and are member of a club. This type hierarchy is depicted on the right hand side of Fig. 3. person: Name' String
I
I Birthdate:Date I
EmplD: Integer i Salary: Integer
Stud D: nteger
Ranking: Integer Club: String
Fig. 3. Distinguishing typing and classification In reality, a person may be classified as a student and a tennis player simultaneously. This means, that a person may play the role of an employee and a tennis player at the same time. In order to support the modelling of such facts, a data model must allow objects to have different types simultaneously. In our example, it must be possible that the same person object can be dressed with the types employee and tennisplayer. Most of the object-oriented data models do not support such role modelling. An object is considered to be an instance of a single type. The next subsection introduces a data model which separates typing from classification and supports role modelling.
3.2
The OM Model
Figure 4 shows a simple schema using the notation provided in the OM model. It models collections P e r s o n s and A d d r e s s e s which are related with each other through association have. Cardinality constraints specify that a person has one or more addresses, while an address may be related with one or more persons. Collection P e r s o n s has three subcollections, namely Employees, S t u d e n t s and T e n n i s P l a y e r s . The integrity constraint c o v e r demands that each person is also classified in at least one of the subcollections, or, in other words, that collections Employees, Students and TennisPlayers cover collection Persons.
121 In the shaded region of each box denoting a collection, the member type of the collection is given. Collection P e r s o n s , for example, has a member type p e r s o n , which means that each object contained in collection P e r s o n s must be dressed with type person.
~nisDlaver
I I Employeesill I Students pITennisPlayers~ Fig. 4. A schema in the OM model
T h e D a t a S t r u c t u r e s . The OM model supports collections of objects and associations. A collection of objects contains objects dressed with the same member type. A collection is itself an object and thus has an object identifier, and it is possible to have collections of collections. Figure 5 shows how role modelling is supported in the OM model. An object may be classified in two different collections simultaneously, for example in a collection Employees and a collection T e n n i s P l a y e r s . Depending on the collection through which we view this object, we see a different representation of it. Looking at an object through collection Employees shows attribute values such as a name, a birthdate and an employment number plus salary as defined in type employee. Viewing the same object through collection T e n n i s P l a y e r s , however, shows attribute values according to type tennisplayer.
Fig. 5. Viewing an object in different roles An association allows objects in collections to be related with each other. An association is a binary collection since it contains pairs of object identifiers where one object identifier refers to an object in a source collection while the other one refers to an object in a target collection. Associations are objects as well and thus have an object identifier.
122
A l g e b r a . The operational model of OM is based on a generic collection algebra. A list of operations supported in OM is given in Table 1. The algebra includes operations such as union, intersection, difference, cross product, flatten, selection, map and reduce as well as special operations over binary collections such as domain, range, inverse, compose and closure. Table 1. Algebra operations supported in the OM model Algebra OperationSignature Union
U : (coilType1, coilType2) --+coilType1 tAType2 n : (collType~, collType~) -+ collTypel n Type, - : (collType~, collType~) ~ ~ollType, • : (collTypeA, coilType2) -+ coll(Type~, Type~) f l a t t e n : collcollType -~ coilType ~ : (~oUType, Type -~ boolean) -+ coUType map: (collTypel, Type1 -~ Type2) ~ coilType2 reduce: (coilType1, (Type,, Type) --+ Type, Type) -+ Type d o m a i n : coll(Typel, Type2) ~ coilType1 r a n g e : coll(Typel, Type2) -+ coilType2 i n v : coll(Typel, Type2) ~ coll(Type2, Type1) o : (coll(Type~, Type2), coll(Type~, Type,)) coll(Typel, Type,) cZosure : coll(Typel, Type~ ) -+ coll(Type~, Type~ )
The algebra operations are given specifying the input arguments and the results along with their types. Collections which list two types, for example, coll(Typel, Type~) are binary collections, containing objects of the form , where oidl refers to an object of type Type1 and oid2 to an object of type Types. For set operations, the notions of least common supertype and greatest common subtype are used to determine the member type of the resulting collections. The union of two collections thus returns a collection whose type is the least common supertype of the two collections involved (Type1 U Type2). The resulting collection contains all the elements belonging to one (or both) of the argument collections. The intersection of two collections, on the other hand, returns a collection whose type is the greatest common subtype of the two collections involved (Type1 7 Types), containing the objects which are members of both argument collections. The type of the resulting collection of the difference of two collections corresponds to the type of the first argument in the operation. The resulting collection contains exactly those objects in the first argument collection which are not also members of the second one. The cross product of two collections returns a binary collection coll(Typel, Types) containing all combinations of an object of the first collection with an object of the second one. The selection operation has as arguments a collection and a function which selects objects from the collection. The result is a subset of
123
the argument collection. The map operator applies a function with a signature Type1 --+ Type2 to each object in the argument collection and returns a collection of objects of type Type2. The flatten operator takes a collection of collections of the same member type and flattens them to a collection of type Type. The reduce operator allows the execution of aggregate functions over a collection of values. It has three arguments - the collection coilType1 over which the aggregate function is executed, the aggregate function itself with a signature (Type1, Type) --+ Type and an initial value of type Type. The OM model also supports special operations over binary collections. Some of them are listed in the second part of Table 1. The domain operation takes a binary collection and forms a collection of all the objects that appear as the first element of a pair of objects < o i d l , old2 > belonging to the binary collection, whereas the range operation returns a collection containing the second elements of such pairs. The inverse operation swaps the first and the second element of the pairs contained in the argument binary collection and returns them in a new binary collection. The composition operation combines those binary objects of the two argument collections where the second object in the first binary object < o i d l , oid2 > appears as first object of the second binary object, . The resulting collection contains binary objects, for example, . The closure of a binary collection is the reflexive transitive closure of a relationship represented by the binary collection. Union, intersection and difference of collections, cross product, compose, range, domain, flatten, inverse, closure simply manipulate object identifiers. Exceptions are the selection, map and reduce operations. A selection operation, for example, might access attribute values of the objects in discourse to select the desired ones.
C o n s t r a i n t s . The OM model supports constraints such as the subcollection, cover, disjoint, intersection and cardinality constraints. The subcollection constraint demands that each object in a collection is also member in its supercollections. The disjoint, cover and intersection constraints over collections restrict the form of relationship between supercollections and their subcollections. The disjoint constraint demands that a set of subcollections do not share a single object of their common supercollection, whereas the cover constraint demands that each object of the supercollection appears in at least one of the subcollections involved in the constraint. The intersection constraint relates a subcollection with its supercollections such that all the common objects of the supercollections are also members of the subcollection. The cardinality constraints are used to restrict how often an object may be contained in an association. These constraints can be evaluated again by simply manipulating object identifiers. The constraint themselves are also objects.
124
4
OMS/Java
OMS/Java (Object Model System for Java) Mes97 can be considered as an object-oriented database management system, respectively, as an object-oriented framework for the Java environment supporting the OM generic data model presented in Sect. 3.2. There exist also other instantiations of the OM model for different environments such as OMS/Prolog NW97 and OMS/Objectivity Pro97. OMS/Java can be characterised by three main features: - The OMS/Java core system is extensible in the form described in Sect. 2.2 - OMS/Java can either serve as an Object-Oriented Database Management System (OODBMS) or as an Application Framework. Used as an OODBMS, OMS/Java can be, for example, a server component in a client/server environment. In the other case, a developer can use the OM model to analyse and design the application domain and then use the OMS/Java framework for implementing the applications. Hence, no mapping is necessary between the resulting design model and the implementation. - Using OMS/Java for managing instances of Java classes is easy and does not cause any major redesign of existing class hierarchies. Figure 6 shows a scenario where several client applications are connected to the same OMS/Java database server.
OM~Java Se~m"
Fig. 6. OMS/Java: a multi-purpose system Note that in Fig. 6 the OMS/Java application framework is part of all client applications, and that the storage management component of the OMS/Java server is exchangeable. This means that one can use as a storage management system either an existing database management system such as Objectivity/DB or a persistant Java system such as ObjectStore PSE for Java. In the following sections, we describe those concepts of the OMS/Java system which are important to understand the basic ideas behind the framework as well as the decisions made to facilitate system extensibility. 4.1
Core
Architecture
The OMS/Java system is divided into three main components:
125
- The Configuration Component The configuration component consists of all those parts of the system that are exchangeable and extensible, including: 9 Object Identifier Manager 9 Algebra Operations 9 Constraints 9 Query Language 9 Data Definition Language 9 Data Manipulation Language 9 Data Structures (such as Hashtables) and Data Access Algorithms - The Core System The core system manages all the data structures needed for supporting the OM model. A description of some of these structures is given in Sect. 4.2. Further, special design patterns have been used to connect the core system to the other components (Sect. 4.3). Finally, all features of the system are available through an Application Programming Interface (API). - The Storage Management Component The storage management component is responsible for not only the persistence of all data, but also for Transaction Management, Concurrency Control, Recovery and Security which includes Access Control and Authentication. Since this component is actually designed in such a way that it can be connected to other database management systems, most of these tasks are passed on those systems. 4.2
Basic Concepts
OMS/Java supports the core object model OM (Sect. 3.2) through the notion of OM Objects, Collections and Constraints. O M O b j e c t s . An object in the OM model does not represent the same entity as a Java object which is an instance of a Java class. An OM object has a unique object identity during its whole lifetime. This differs from the way object references are handled in Java. A reference to an object in Java is only unique as long as the object is loaded in the Java Virtual Machine. Consider, for example, that you store a Java object in a file and then read it back again from that file. Reading from the file means that the Java Virtual Machine creates a new Java object with a new reference to it so it is not possible to decide whether the object you stored before is the same as the one you retrieved. To solve this problem, we introduced a special class ObjectID. Every instance of this class represents an unique persistent object identifier and is related to exactly one OM object. Further, depending on the context in which an object is accessed (e.g. accessing an object through a specific collection), the object changes its role. This implies that more than one type can be associated with an OM object. This is not supported by the Java environment since it is not possible to change the type definition of a Java class instance during run-time.
126
As is shown in Fig. 7, we solved this problem by splitting up an OM object into the two Java classes 0H0bjecC and 0NInstance (which is actually a Java interface class). Additionally, meta information about OM objects needed by the OMS/Java system are defined in instances of the Java class 0NType.
Fig. 7. OM Objects Another approach would have been to implement one's own dynamic type system as has been done, for example, in the OMS/Prolog system [NW97]. But, since we wanted to be able to use standard Java classes in our system, this approach was not suitable. Using Java classes involves two steps: first the meta information has to be defined in the schema (see also Sect. 5.2) such as type (
person
name:
String;
); Then, for all the types defined in the schema, a corresponding Java class has to be specified. This can be done in three ways: - by extending the class 0MSimpleInstance which provides all functionality needed by the system such as meta information. by implementing the interface 0HInstance, or - by using the adapter class 0HAdapter. An adapter or wrapper "converts the interface of a class into another interface which clients expect. It lets classes work together that could not otherwise because of incompatible interfaces." [GHJV95]. Hence, the 0HAdapter class makes it possible to use most of the Java classes without modifications. The following class definition is an example for the first approach: package demo;
import omJava.*; public class Person extends OMSimpleInstance { public String name; )
127
C o l l e c t i o n s . An OM object can be classified according to different criteria by inserting it into collections. Removing an object from a collection and inserting it into another one simply changes its role. So, if an OM object is stored in a collection then it gains automatically the member type of that collection. Further, a collection itself is an OM object and can therefore also be classified by inserting it into other collections. A rich set of algebra operations over collections (see Sect. 3.2) together with different kinds of collections such as sets, bags and sequences are also provided by the OMS/Java system.
Constraints. The various structural constraints such as subcollection, cover, disjoint and intersection between collections (Sect. 3.2), as well as the cardinality constraints for associations, are specified as Global Constraints in OMS/Java and will be checked at commit time of a transaction. Local Constraints, on the other hand, are related to a single OM object. For example, the constraint that all objects in a collection must be of the same type is defined as a Local Constraint and will be checked every time the collection is updated. Local Constraints are in fact specialisations of Triggers. A Trigger is related to one or more OM objects and is activated as soon as the trigger condition holds. Note that Global Constraints, Triggers and Local Constraints are also treated as OM objects in OMS/Java.
4.3
D e s i g n Issues
To take full advantage of object-oriented concepts such as reuse and extensibility, it is important to use the appropriate design patterns. "Each pattern describes a problem which occurs over and over again in our Environment, and then describes the core of the solution to that problem, in such a way that you can use this solution a million times over, without ever doing it the same way twice." GHJV95 Using design patterns not only increases the quality of the software product, but also makes it possible to build systems that are flexible enough to support requirements not known while implementing it. The following four design patterns were used to achieve extensibility of our system:
A b s t r a c t F a c t o r y . The main construct for system parameterisation, respectively, configuration is the Abstract Factory. It defines a set of abstract parameters needed at run-time by an application or, in our case, the OMS/Java core system. Such parameters include among others the algebra used for operations over collections, the object identifier generator, the query language and data definition language parsers and the various data structures such as hashtables (Sect. 4.4).
128
A b s t r a c t Classes. An Abstract Class defines a common interface for its subclasses. It defers some or all of the implementation to the subclasses. Hence, an abstract class cannot be instantiated, but forces the developer of the subclasses to at least implement important methods needed for the overall system functionality. For instance, there is a data structure C o n t a i n e r which is used to store the elements of a collection. Inserting elements into such a container depends on whether a collection is a set, bag or sequence. For this purpose the abstract class InsertionStrategy defines a method insert which provides a c o m m o n interface for inserting objects into containers and which must be specified by its subclasses: abstract public class InsertionStrategy { , . ,
abstract public void insert(Container container, Object object);
}
S t r a t e g y . The Strategy design pattern defines a family of algorithms. By encapsulating all these algorithms into a strategy object, they become interchangeable and independent from a specific client object. For example, the algorithms to insert objects into collections are defined as insertion strategies making it possible to vary the insertion algorithm depending on whether a collection is a set, a bag or a sequence. public class SetStrategy extends InsertionStrategy { public void insert(Container container, Object object) { ... }
} public class BagStrategy extends InsertionStrate~ { public void insert(Container container, Object object) { ... }
}
Bridge. "A Bridge design pattern decouples an abstraction from its implementation so that the two can vary independently." GHJV95 For example, two bridges have been used to separate the algebra operations and the underlying data structures from the collection object. Therefore, it is possible to exchange the data structures and/or the algebra without having to modify the interface of the collection object. This implies that the collection object must be initialised by these two bridges. The following example illustrates the use of these bridge classes. Suppose we want to find all objects of which the attribute name is equal to the value Smith using the algebra operation select: OMCollection result - collection.select("name", "Smith");
129
This invokes the method s e l e c t of the OMCollection object which calls the corresponding method of the Algebra object. This method then performs the selection and inserts the matching objects into the result collection by invoking the i n s e r t method of this collection. That method again redirects the insertion by calling the i n s e r t method of the I n s e r t i o n S t r a t e g y object which itself invokes the method i n s e r t of the Container object. public class OMCollection extends OMSimpleInstance { protected Algebra algebra; // decouples algebra protected Container container; // decouples data structure protected OMInsertionStrategy strategy; public OMCollection select (String selection, Object value) { return algebra.select (this, selection, value) ;
} public OMCollection insert(Object object) { strategy.insert(container, object); }
public class OMAlgebra implements Algebra { public OMCollection select(OMCollection collection, String selection, Object value) { OMCollection result = ... // perform select operation over // 'collection ~ result.insert(object.key()); // insert matching objects into //result collection return result;
} public class Container { public void insert(Object object) {...}
} T h e Link b e t w e e n t h e Core S y s t e m a n d t h e C o n f i g u r a t i o n Component
4.4
The CoreSystem is connected to the Configuration Component by an instance of the Java class Factory as shown in Fig. 8. This F a c t o r y class corresponds to the design pattern Factory and is an important concept for making a system extensible. The F a c t o r y class provides various methods such as getNextObj e ct ID (), getOMLparser(), getCollectionAlgebra() and getCollectionContainer(). The method getNextObjectID(), for instance, returns an instance of class Objec~ID which represents a unique object identifier, whereas g e t C o l l e c t i o n
130
Algebra() returns an instance of Algebra which can be used for evaluating algebra operations on collections. All parts of the Configuration Component are obtained by calling the appropriate method of the Factory instance which, for its part, is given as a parameter while initialising the Core System. By subclassing the class Factory and overriding methods such as getNext0bjectID(), it becomes possible to provide to the core system, for instance, different data structures such as object identifiers.
Fig. 8. The link between the core system and the configuration component
5
From
OMS/Java
to TOMS/Java
Extending OMS/Java with new features means in fact to extend, respectively, to replace parts of the Configuration Component which has been described in Sect. 4.1. The following list gives a summary of the main tasks that are involved when implementing extensions to OMS/Java: - Extending the syntax of the Data Definition Language, the Data Manipulation Language and Query Language, and changing the corresponding parsers, i.e. implementing the new features in subclasses of the standard parsers. - Subclassing the standard object identifier class Implementing new algebra operations in a subclass of the algebra class Extending the various constraint classes - Defining a subclass of the class Factory which is used as a link between the Core System and the Configuration Component -
-
The following sections describe in more detail how OMS/Java has been extended to support the Temporal Object Data Model TOM SN97c, SN97a. 5.1
The Temporal Object Data Model TOM
TOM is based on the generic object data model OM Nor93 and exhibits many of the features found in various temporal object-oriented models, e. g. RSgl,
131
WD93, KS92, BFG96, but in a more generalized form. For example, we timestamp not only data values, but also collections of objects and the associations containing the temporal relationships between objects. Anything considered to be an object in our model may be timestamped, even metadata such as constraints, types or databases. This allows the specification of a lifespan for each instance of such a construct. Hence, TOM is based on object-timestamping, i.e. the object identifiers are extended with a timestamp. A timestamp is a temporal element Gad86 - a set of time intervals, closed at the lower bound and open at the upper bound. Additionally, TOM supports a temporally complete query lanaguage BJS95 and temporal constraints. Temporally complete means that all operations found in the non-temporal data model have to be generalized into temporal operations, including, for example, the usually neglected set difference operation. Additionally, temporal comparison predicates have to be supported allowing expressions such as a before b. Temporal constraints generalise their non-temporal counterparts with temporal semantics. A constraint no longer has to hold over a single database state, but rather over the set of those database states contained in the constraint's lifespan. 5.2
Language Extensions
This section describes the extensions made to the syntax of the various languages (DDL, DML and QL) by means of an example: Suppose we want to build an information system based on the schema given in Fig. 4. For the sake of simplicity, we model only the collections Persons, Employees and Students. Defining t h e Schema. The following schema has been specified using OML, the data definition language of OMS/JAVA: person = demo.Person; // links type 'person' to the Java class // 'demo.Person'
employee = demo.Employee; student
ffidemo.Student;
type person
lifespan{1980/1/1-inf)}
( name:
String;
); type employee subtype of person lifespan{1980/I/1-inf)} ( empID: Integer; salary: Integer; );
132 type s t u d e n t subtype of p e r s o n l i f e s p a n {1982/1/1-inf)}
( studID: I n t e g e r ;
); collection Persons collection Employees collection Students
: s e t of person lifespan {1980/l/l-inf)}; : set of employee lifespan {1980/1/1-inf)}; : set of student lifespan {1982/1/1-inf)};
constraint: Employees subcollection of Persons lifespan
{ 1980/1/1-inf) }; c o n s t r a i n t : S t u d e n t s s u b c o l l e c t i o n of Persons lifespan
{ 1982/1/1-inf) }; Three categories of objects are defined in this schema: type, collection and constraint objects. Further, the Lifespan of TOM objects, specifying the period of time in which an object is valid, is set by the corresponding lifespan parameter. For supporting the notion of Lifespans, we first had to extend the syntax of OML by the following definitions given in EBNF Wir96: lifespan = "{" interval {"," interval} "}". interval = "" date "-" date ")". date = (year "/" month "/" day "~" hour ":" minute ":" second) ~ "inf".
Next, the OML parser had to be adapted to the new syntax. Since all parsers in our system are part of the Configuartion Component and not of the Core System, no changes are necessary in the Core System classes for supporting, for example, a new syntax. Only the corresponding parser has to be changed which can be achieved by either replacing the existing parser or by extending it. In most cases, especially if there are only minor changes of the syntax, extending an existing parser will be possible because we designed the standard parsers in such a way that they invoke special methods during the syntax analysis. For example, after the OML parser has parsed a type, collection or constraint definition such as type person, the method parseRestOfLine(...) is called. In the case of the standard parser, this method just returns without performing any action. Hence, for supporting the new Lifespan syntax we just extended the Java class OHLparser and overrode the parseRestOfLine(... ) method. C r e a t i n g Objects. One possibility for creating objects and inserting them into collections is to import a file in which the data is described in terms of Data Manipulation Language statements: c r e a t e o b j e c t paul lifespan {1969/1/1-inf)} dress object paul as demo.Person during {1969/1/1-197S/6/1)} values (name = "Paul Jones")
133
dress object paul as demo.Employee during {1986-1996)} values (empID = 23; salary = 2000) dress object paul as demo.Student during {1993-1995)} values (studID = 2674)
i n s e r t object paul into c o l l e c t i o n Persons during {1985/1/1-1996/1/1)} The c r e a t e o b j e c t statement defines an OM Object the alias of which is paul. Its lifespan is set to the Lifespan parameter l i f e s p a n 1 9 6 9 / 1 / 1 - i n f ) . The d r e s s o b j e c t statements further specify that instances of the Java classes demo. Person, demo.Employee and demo.Student are associated to that object and that this association is valid as given by the During parameters. Remember that an OM Object can refer to more than one instance of a Java class (as shown in Fig. 7). The i n s e r t statement states that the object paul is a member of the collection Persons during the time period specified by the During parameter duri ng 19851/1-19961/1). Note that the collection Persons has been already created after loading the schema. To support the Lifespan as well as the During parameters, which are features of TOM, we had to extend the DML Parser taking the same approach as in the case of the OML Parser, i.e. we extended the DML Parser by making a subclass of the class DMLparser and implemented, respectively, overrode some methods.
Q u e r y i n g t h e D a t a b a s e . AQL (Algebraic Query Language) is the query language of OMS/Java. This language was originally developed for OMS/Prolog NW97, and it is both a simple, and yet powerful language with operations over values, objects, collections and associations. As a simple example, consider that we want to know which objects are classified as both Employees and Students. This can be expressed in AQL as: Employees union Students; In the case of TOM the corresponding query contains the keyword v a l i d in addition: valid Employees union Students;
The v a l i d keyword denotes that the query should be evaluated only on those objects which are valid now. The result of this query is presented by TOMS/Java in the form of a scrollable list. By double-clicking on one of the items in the list, the system dynamically generates a window showing all properties of the selected object. Figure 9 gives an example of such an object. As in the case of DDL and DML, we have extended the Java class AQLparser, in this case to support temporal queries.
134
Fig. 9. A person object
5.3
Changes Made in the Configuration C o m p o n e n t
In this section, we outline the changes made in the Configuration Component that have been necessary to support TOM. Table 2 gives an overview of the classes which had to be extended. Instances of these classes can be obtained by calling the corresponding method of the Factory. Note that the term Factory is used in the text as a placeholder for an instance of the class ValidTimeFactory. This class is a subclass of the OMS/Java class F a c t o r y and overrides methods such as g e t N e x t 0 b j e c t I D ( ) to support the semantic of TOM (Sect. 4.3). Table 2. Java dasses which had to be extended Factory Method OMS/Java class T O M S / J a v a class get0MLparser() ONLparser TOMLparser DMLparser getDMLparserO TDMLparser AQLparser TAQLparser getAQLparserO ObjectID getNext0bjectID() ValidTimeObjectID
O b j e c t Identifiers. When a schema definition file is loaded, the core system first invokes the method get0MLparser() of the Factory to retrieve an instance of an OML parser, followed by a call to the parser method g e n e r a t e ( s c h e m a ) which parses the schema and generates the objects. In the case of TOM, this parser must be able to manage temporal information such as Lifespans. Hence, we made a subclass of 0MLparser containing new methods for temporal information processing as described in Sect. 5.2. For the schema given in Sect. 5.2, the parser creates the following Java instances: instances of 0MType for each type definition, instances of 0MCollection for each collection and instances of 0MSubcollConstr for each subcollection constraint. For each of these instances, the parser also gets an 0bjectID instance, respectively in the case of TOM, a ValidTime0bjectID instance by calling the Factory method g e t N e x t 0 b j e c t I D ( ) . The class ValidTimeObjectID is a subclass of 0bjectID which has been extended for storing Lifespan information. Finally, the parser sets the lifespans of these object identifiers to the ones given in the schema.
135
O b j e c t C r e a t i o n . One way of creating objects is to import a file containing data defined by Data Manipulation Language statements as has been described in Sect. 5.2. For this purpose, the core system calls the Factory method getDMLparser() which returns an instance of the class DMLparser. Actually for TOM, the Factory returns a subclass of DMLparser which provides all methods needed for managing temporal information such as Lifespans (Sect.5.2). The method generate ( i n p u t F i l e ) of the parser is invoked by the core system which imports, respectively, creates the objects. For our example, the parser first creates an 0MObject instance, the alias of which is paul, and then associates it to an instance of ValidTime0bjectID of which the lifespan is set to 1 9 6 9 / 1 / 1 - i n f ) . Further, the parser creates instances of the Java classes demo. Person, demo. Employee and demo. Student and sets the values of the different instance fields by calling the s e t A t t r i b ( . . . ) method of each instance. Finally, the parser relates these instances to the object paul by calling the method dress ( i n s t a n c e ) which inserts an instance into the Instance List of the object paul. The data structure Instance List, which is obtained by the Factory method getInstanceList () while creating an OM Object, is used to manage all OM Instances related to this object and provides methods such as i n s e r t (0MInstance
instance), delete(OMInstance instance), update(OMInstance instance) and replace(OMInstance instance, int index). For supporting TOM, we had to extend the S t d I n s t a n c e L i s t class and to override these methods. In our example, the Validity of the relationship between the object paul and the associated instances of demo. Person, demo. Employee and demo. Student is specified by the During parameters. These parameters are stored by the parser in the corresponding instance by calling the method setParameter(paremeter) of the 0MInstance class. Note that this parameter is of type 0bj e c t making it possible to store any kind of information in an OM Instance object. Collections a n d A l g e b r a O p e r a t i o n s . Collections are represented by instances of the Java class 0MCollection and are created, for example, while loading a schema definition file by the OML parser. Creating such an object invokes the constructor of the 0MCollection class. This constructor itself calls the Factory methods g e t C o l l e c t i o n a o n t a i n e r ( ) which returns an instance of the class Container needed for managing the members of a collection, and g e t C o l l e c t i o n A l g e b r a ( ) which returns, in the case of TOM, an instance of the class ValidTimeAlgebra providing algebra operations such as u n i o n ( . . . ). Since the data structure as well as the algebra are returned by the Factory and are therefore not part of the 0MCollection class which belongs to the core system, it is possible to exchange, for instance, the algebra without having to change the OMCollection class. Inserting objects into collections can be done, for example, by importing a data file using the DML parser. In our example, the object paul is inserted into the collection Persons. In addition, the during parameter, which is a feature of TOM, specifies the time period during which paul is a member of Persons. No changes were necessary to support this feature because an 0MCollection
136
class provides a method s e t P a r a m e t e r ( o b j e c t , parameter) which can be used to store additional information for each member object such as the Validity given by the during parameter. This information is needed by the algebra class and can be retrieved by calling the collection method g e t P a r a m e t e r ( 0 b j e c t o b j e c t ) . Hence, the 0MCollection class does not need to be able to manage this information and since the paramel:er is of type Object, the stored information is not restricted to specific semantics such as temporal information. So, for inserting the object paul into Persons the DML parser first invokes the method i n s e r t ( o b j e c t ) of the collection to insert the object and then sets the Validity given by the during parameter by calling the collection method setParameter(object, parameter). Evaluating algebra operations simply means calling the appropriate methods of collection objects. For example, the AQL parser evaluates the temporal AQL query " v a l i d Employees union Students ;" by invoking the u n i o n ( . . . ) method of the Employees collection object:
employees.union(students, algebraParameter); The method union takes two input parameters: the second collection object for the union operation and an algebra parameter which is in our case the keyword v a l i d . Further, this method calls the method union(employees, students, algebraParameter) of the algebra instance which has been specified when the collection object has been created. Hence, only class Algebra, or subclasses of it such as ValidTimeAlgebra, must know how to handle the algebra parameter. Note that this parameter can also be of any type. C o n s t r a i n t s . Global Constraints, Triggers and Local Constraints, which have been defined in Sect. 4.2, are treated as OM Objects and are therefore associated to an unique object identifier which in the case of TOM is an instance of YalidTimeObj ect ID. Global Constraints such as the subcollection constraint Employees subcollection of Persons lifespan 1980/1/1-inf) in our example are created by the OML parser after loading the schema definition file. The parser invokes for this purpose the Factory method g e t S u b c o l l C o n s t r ( ) which returns, in the case of TOM, an instance of 0MValidTimeSubcollConstr. The Lifespan parameters denote the time periods in which the constraints are valid. The OMValidTimeSubcollConstr class is a subclass of OMSubcollConstr of which various methods have been overridden for supporting temporal semantics. Triggers and Local Constraints are created at the same time when the objects to which they belong are initialised.
6
Conclusions
We presented a general approach for extending database management systems (DBMS) in terms of model extensibility and described the corresponding general
137
DBMS architecture. While our approach builds on ideas of architecture extensibility, i.e. providing architectural support in terms of services such as storage management, it also differs from it in terms of focussing on the generalisation of a core data model and system for advanced application domains. In particular, we described the OMS/Java system which has been developed to support model extensibilty. As a proof of concept, we have extended OMS/Java to support the temporal object data model TOM. We are convinced that using the model extensibility approach leads to database management systems which are easily adaptable to various application domains. In the future, we want to investigate extending the system for a spatial data model and also for a version model. The latter is intended to provide an OMS/Java implementation platform for a document management system currently under development.
References BFG96
E. Bertino, E. Ferrari, and G. Guerrini. A Formal Temporal ObjectOriented Data Model. In P. Apers, M. Bouzeghoub, and G. Gardarin, editors, Advances in Database Technology, pages 342-356. Springer, 1996. BJS95 M. BShlen, C. Jensen, and R. Snodgrass. Evaluating the completeness of TSQL2. In J. Clifford and A. Tuzhilin, editors, Recent Advances in Temporal Databases, pages 153-172, 1995. BLW88 D.S. Batory, T.Y. Leung, and T.E. Wise. Implementation Concepts for an Extensible Data Model and Data Language. ACM TODS, 13:231-262, September 1988. CDG+89 M. J. Carey, D. J. DeWitt, G. Graefe, D. M. Haight, J.E. Richardson, D. T. Schuh, E. J. Shekita, and S. Vandenburg. The EXODUS Extensible DBMS Project: An Overview. In S. Zdonik and D. Maier, editors, Readings in Object-Oriented Database Systems. Morgan-Kaufmann, 1989. CDKK85 H.-T. Chou, D.J. DeWitt, R.H. Katz, and A.C. Klug. Design and Implementation of the Wisconsin Storage System. Software Practice and Experience, 1985. Cha96 A. Chatterjee. A Framework for Object Matching in Federated Databases and its Implementation. Journal of Intelligent and Cooperative Information Systems, 1996. EN94 R. Elmasri and S. Navathe. Fundamentals of Database Systems. Benjamin/Cummings Publishing Company, 1994. Gad86 S.K. Gadia. Toward a Multihomogeneous Model for a Temporal Database. In Proceedings of the International Conference on Data Engineering, pages 390-397, 1986. GD94 A. Geppert and K. Dittrich. Constructing the Next 100 Database Management Systems: Like the Handyman or Like the Engineer. A CM SIGMOD Record, 1994. GHJV95 E. Gamma, R. Helm, R. Johnson, and J. Vlissides. Design Patterns, Elements of Reusable Object-Oriented Software. Addison-Wesley, 1995. HCL+90 L.M. Haas, W. Chang, G.H. Lohman, J. McPherson, P.F. Wilms, G. Lapis, B. Lindsay, H. Pirahesh, M.J. Carey, and E. Shekita. Starburst Mid-Flight: As the Dust Clears. IEEE Trans. on Knowledge and Data Engineering, 1990.
138
HS951 KS92 MBH +
Mes97 Nor93
Nor95
NW97
Pro97 RS91
SC94
SN97a
SN97b
SN97c
WD9~
Wir96
A. Heuer and G. Saake. Datenbanken: Konzepte und Sprachen. International Thomson Publishing, 1995. W. KMer and H. SchSning. Realizing a Temporal Complex-Object Data Model. In SIGMOD Conference 1992, pages 266-275, 1992. F. Maryanski, J. Bedell, S. Hoelscher, S. Hong, L. McDonald, J. Peckham, and D. Stock. The data model compiler: A tool for generating objectoirneted database systems. In Proc. of the 1986 Intl. Workshop on ObjectOriented Database Systems. S. Messmer. The OM Model as a Framework for the Java Environment. Master's thesis, Department of Computer Science, ETH Zurich, 1997. M. C. Norrie. An Extended Entity-Relationship Approach to Data Management in Object-Oriented Systems. In Proceedings of the 12th International Conference on the ER Approach, 1993. M. C. Norrie. Distinguishing Typing and Classification in Object Data Models. In Information Modelling and Knowledge Bases, volume VI, chapter 25. lOS, 1995. (originally appeared in Proc. European-Japanese Seminar on Information and Knowledge Modelling, Stockholm, Sweden, June 1994). M. C. Norrie and A. Wiirgler. OMS Object-Oriented Data Management System. Technical report, Institute for Information Systems, ETH Zurich, CH-8092 Zurich, Switzerland, 1997. R. Prodan. OMS/Objectivity. Master's thesis, Department of Computer Science, ETH Zurich, 1997. E. Rose and A. Segev. TOODM - A Temporal Object-Oriented Data Model with Temporal Constraints. In Proceedings of the l Oth International Conference on the ER Approach, 1991. A. Segev and A. Chatterjee. Supporting Statistical Operations in Extensible Databases: A Case Study. In Proc. IEEE Seventh International Working Conference on Scientific and Statistical Database Management, Charlottesville, USA, sep 1994. A. Steiner and M. C. Norrie. A Temporal Extension to a Generic Object Data Model. Technical Report 265, Institute for Information Systems, ETH Z/irich, May 1997. A. Steiner and M. C. Norrie. Implementing Temporal Databases in ObjectOriented Systems. In Database Systems for Advanced Applications (DASFAA), 1997. A. Steiner and M. C. Norrie. Temporal Object Role Modelling. In Proceedings of the Conference on Advanced Information Systems Engineering (CAiSE}, 1997. G.T.J. Wuu and U. Dayal. A Uniform Model for Temporal and Versioned Object-Oriented Databases. In A. Tansel, J. Clifford, S. Gadia, S. Jajodia, A. Segev, and R. Snodgrass, editors, Temporal Databases: Theory, Design, and Implementation, chapter 10, pages 230-247. Benjamin/Cummings Publishing Company, 1993. N. Wirth. Compiler Construction. Addison-Wesley, 1 edition, 1996.
An Environment for Designing Exceptions in Workflows* F. Casati, M.G. Fugini, I. Mirbel {casati,fugini,mirbel} @elet.polimi.it Politecnico di Milano Piazza L. Da Vinci, 32 - 1-20133 Milano, Italy A b s t r a c t . When designing a WorkFlow schema, often exceptional situations, such as abnormal termination or suspension of task execution, have to be dealt with by the designer explicitly, as well as operations which typically occur at the beginning and termination of tasks. This paper shows how the designer can be supported by tools allowing him to capture exceptional behavior within a WF schema by reusing an available set of pre-configured exceptions skeletons. Exceptions are expressed by means of triggers, to be executed on the top of an active database environment; triggers can autonomously treat exceptional events or typical start/termination situations arising in a WF. In particular, the paper deals with the treatment of typical WF exceptional situations which are modeled as general exception skeletons to be included in a new WF schema by simply specializing or instantiating them. Such skeletons, called patterns, are stored in a catalog; the paper describes the catalog structure and its management tools constituting an integrated environment for pattern-based exception design and reuse.
1
Introduction
WorkFlow (WF) systems are currently becoming a more and more applied technology in the direction of providing a common framework for supporting activities within office information system. WFs in the organizations are modeled to capture the possible sequences of activities needed to reach a given goal, and to provide tools and links to the existing Information Systems of the organization. Many aspects have to be considered in conjunction with the pure activity flow. In particular, the documents and information associated with the activities, the performers of the activities and their role in the organization, and the possible exceptions to the normal flow of activities. In other related fields, such as in Software Engineering and Information Systems, analogous techniques and tools are used combining data and function modeling to represent the application functionalities and to represent b o t h normal and abnormal situations and exceptions14. Currently, few techniques and tools exist for W F design. Some of them 17 are based on rules and catalogs of predefined W F portions for W F management. * Research presented in this paper is sponsored by the WIDE Esprit Project N. 20280
140
In general, the designer of a WF schema has to provide many details in order to obtain a schema which is executable by a WorkFlow Management System (WFMS). This description requires the specification of both the normal flow and of the possible variations due to exceptional situations that can be anticipated and monitored. This paper addresses the issue of providing support in designing exceptions in a WF focusing on exceptions which can be statically foreseen. It describes an e n v i r o n m e n t of tools for the specification of WF schemas. Exceptions can be included in a schema by reusing (i.e., selecting and customizing) a set of pre-defined generic exceptions. The term "exception" is used in a broader sense than simply for exceptional situations. In fact, exceptions model also conditions regarding the starting, termination, or repetitions of WF instances (called cases) or regarding the periodical occurrence of operations in the WF execution. The support environment, called WERDE, consists of a catalog of pre-designed WF schema portions, and of tools for accessing the catalog. We are mainly concerned with the specification of exceptional situations in WFs, that is, situations where events occuring in a WF schema may cause a task to be canceled, or a WF case to be terminated. Such exceptional situations may be designed in advance and inserted in the WF schema specification by including exception handlers for the most likely occurring events or by designing typical termination handling strategies. Another situation is the WF start phase which can be subject to typical situations. For instance, a WF has to be started periodically, or at a given time. These situations may be included as a pre-set part of the schema specification. The catalog of the environment stores WF schema portions, mainly dealing with exceptions, to be reused during the design of a new schema. The contents of the catalog is a set o f typical patterns16 enabling the designer to deal with frequent occurrences of similar situations in WF design in given application domains, and with generic situations that may occur in any application domain. The catalog, and its related design support tools, is based on the following main issues: - rules: rules model exceptions, starting and termination situations, and WF
schema evolution due, for instance, to changed requirements deriving from new types of exceptional events; - patterns: typical rules may be used in several designs. Patterns described in the paper are rules, or sets of related rules, to control a given typical situation. Patterns are a way for the designer to specify the WF schema by reusing previous design experience. Rules in WF modeling have been considered, among others, by 12. However, they are mainly used to represent the complete flow. In other approaches to WF modeling (13, 15), a strong emphasis is put on providing a simple and graphic interface to WF design. However, if such interfaces are to capture all the details of anomalous flows, readability is penalized, such as in Petri Net based approaches (2).
141
The approach of this paper based on pattern reuse aims at improving the speed and quality of WF schema design. A pattern is a generalized description of a rule or sets of rules that can be associated to a WF schema, either to model exceptions, or to model starting and termination situations, or situations to be controlled due to schema evolution. Reuse, also based on patterns, is beeing successfully applied to software development (5, 18, 19). The paper focuses on the functionalities of the W E R D E environment which allow the designer to reuse patterns stored in a catalog for dealing with exceptions in WF schema design. Functionalities for pattern design are also presented. The W E R D E environment is presented mainly in terms of specifications of its functionalities; then, the implemented functionalities are presented in detail. The paper is organized as follows. In Section 2, we illustrate the catalog of patterns, and describe the use of rules in WF design by presenting exceptions and patterns. In Section 3, we present the functionalities of the environment for access and management of the catalog and show the tools usage for exception design in WF schemas and for exception analysis. Finally, in Section 4, we comment and give some hints about future work.
2
A Pattern
Catalog
This section describes the pattern catalog of the WERDE environment where a set of pre-defined WF patterns is stored at different levels of abstraction and genericity in a layered structure. The catalog is the centralized repository of reusable WF elements in order to support specification by example of different elements of a WF, at different granularity levels, such as process variables, exceptions, and tasks. Exceptions and patterns will be described in the following of this section. Patterns model situations that can occur in WF design in an applicationindependent way, at different levels of genericity (e.g., integrity constraints holding for schemas of any application domain, or schema fragments describing frequently occurring combinations of tasks in a given application domain). They can be described either using a language for specifying exceptions and/or through the WF process description language WPDL 6. In Figure 1 the categories of patterns stored in the catalog are depicted. Patterns are classified at three different layers. -
B u i l t - i n p a t t e r n s . Patterns at this level deal with reaction to events and time alarms (Basic patterns), integrity and security constraints over WF data (Authorization, Integrity, and Count patterns), WF management (Starting, Interaction, and Continuation patterns), and with changes in the organization structure (Organization and Schema evolution patterns). They describe context-dependent and time-dependent knowledge related to basic exceptional situations independent of the semantics of all possible applications. Built-in patterns specify, for example, basic interactions between WF elements, or situations where a task has to be suspended or resumed, or express
142
Pattern catalog Integrity patterns Continuation patterns Count p a t t e m ~ Basic patterns Built-in patterns
Interaction patterns Starting patterns Authorization p a ~ Schema-evolution patterns Organizational patterns Other patterns
Genetic
Information use Remainder Other pattems
Generic application-oriented structures
Document preparation2p_....._ Document revision............ Reservation Other patterns
Fig. 1. The WERDEpattern catalog exception requirements related to WF interoperability. Built-in patterns are implemented as generic rules that can be adapted to different application contexts. G e n e r i c tasks. Patterns at the generic tasks layer allow the description of frequently occurring activities in WF design in the form of tasks, independent of the documents involved in their execution (e.g., a task for registering information in a document). - G e n e r i c application-oriented structures. This layer of the catalog contains WF fragments and associated exceptions needed to model frequently occurring situations of one or more application domains. These are frequently occurring combinations of tasks (i.e., WF fragments) generally performed together to achieve an organization goal in a given application domain (e.g., a generic structure describing the concurrent review of a document by several people).
-
In order to be accessible and user friendly for the designer, the catalog contains both a Pattern Specification element and a Sample Usage element. The first one describes the exception and includes parts allowing the designer to understand the goal of the pattern, to locate the pattern in the catalog, and to link patterns together within a WF schema. The second element contains sample instantiations of patterns in various application domains, thus guiding the designer in completing a new schema.
143
In the following of the paper, we describe exceptions and patterns. In particular, we show how exceptions are specified and used, and how exceptions are then abstracted in patterns. For details about the patterns contained in the various categories of the catalog, the interested reader can refer to 9.
2.1
Exceptions
A short introduction to the exception language is now given (see 7 for a detailed description). The Chimera-Exc exception language is based on the Chimera language for active object-oriented databases. An exception is specified by an eventcondition-action (ECA) rule (also called trigger in the following). Events denote the occurrence of a possibly exceptional situation; conditions determine if the occurred event actually corresponds to an exceptional situation to be managed, while the action part defines the operations to be performed in order to manage the exception. ECA rules may refer in their event, condition, or action part, to objects and classes of the WFMS database. These include static and dynamic information on WFs and tasks (stored in objects of classes case and task respectively), data relating W F participants (objects of class agent), plus a WFspecific class, bearing the name of the WF, whose objects store process-specific variables local to each single case. For instance, variables needed for the execution of cases of a hotel r o o m R e s e r v a t i o n WF, such as the room number or the customer name, are stored as attributes of an object of the r o o m R e s e r v a t i o n class. An exception can be sensitive to different kinds of e v e n t s : data events are raised upon modification of the content of the WFMS database. Data events include the creation of a new object (create), the modification of an object's attribute (modify), and the deletion of an object (delete). For instance, event m o d i f y ( t a s k ) is raised when an attribute of an object of the task class is modified. Workflow events are raised when a case or task is started or completed. W F events include c a s e S t a r t , caseEnd, t a s k S t a r t , and taskEnd. For instance, event t a s k S t a r t (rayTask) is raised when an instance of task myTask is started. Temporal events are raised upon the occurrence of a defined temporal instant. They can be expressed as deadlines, temporal periods, or interval elapsed since a specific date or time. For instance, event e l a p s e d 1 day since t a s k S t a r t (myTask) is raised as one day has elapsed since the start of an instance of task myTask. Finally, external events, qualified by their name, are notified by external applications: these may correspond to applicative situations, such as a phone call by a customer cancelling a reserved room. An example of external event is raise (cancelReservation).
The condition part consists of a declarative query, expressed by a formula evaluated over the WFMS database state. The formula is expressed as a conjunction of predicates, and includes a set of variables to which bindings are associated as a result of the formula evaluation: if the result of the query is empty (i.e., if no bindings are produced), then the condition is not satisfied, and the action part is not executed. Bindings resulting from the formula evaluation are passed to the action part in order to perform the reaction over the appropriate objects.
144
The condition includes class formulas, for declaring variables ranging over the objects of a specific class (e.g., task(T)), type formulas for declaring variables of a given type (e.g., r e a l (R)), and formulas expressing comparisons between expressions (e.g., T. executor--"Lisa"). Objects affected by events are captured in the condition part by the occurred predicate. Thus, for instance, in an exception triggered by the c a s e S t a r t event, the binding with the started instance is obtained by means of the predicates case (C), occurred ( c a s e S t a r t , C). The a c t i o n part includes data manipulation actions, updating the content of the WFMS database, and operations executed through calls to the WF engine. The first category includes the primitives create, delete, and modify. For instance, d e l e t e ( L o g , L ) deletes all objects of class Log to which L is bound after the condition evaluation. The second category includes primitives to start, rollback, or suspend the execution of cases and tasks, assign or delegate the execution of tasks, and send messages to agents. Examples are delegateTask (T, A), startCase (mySchema, start ingParameter), and not ify (T. executor, "deadline is approaching"). A sample trigger activated at the end of cases of the roomReservation W F is as follows. The condition determines which isthe case that has caused the triggering (predicates case (C), occurred (caseEnd, C)), retrieves the object storing the variables of the just completed case (roomReservation(R), R. caseId=C) and then checks if the customer has not paid (R. paymentDone=FALSE). If the condition is verified (i.e.,if bindings are produced), then the action part is executed, causing the start of an instance of the cancelBooking N F and providing the customerID as input parameter. d e f i n e t r i g g e r missingPayment events caseEnd condition case(C), occurred(caseEnd,C), roomReservation(R), R.caseld=C, R.paymentDone=FhLSE actions startCase("cancelBooking",R.customerId) end
Rules can be associated to a task, to a schema, or to the whole WFMS. Tasklevel rules model exceptions which are related with a single task; an example is a rule reacting to the task deadline expiration. WF-level rules defines a behavior common to every task in a schema, or model exceptions related to the whole schema; for instance, a WF-level rule could specify that all tasks of a the WF should be suspended at 7pm. Finally, global (WFMS-level) rules model behaviors defined for every task or case, or exceptions involving cases of different WFs. For instance, an exception stating that "tasks that were assigned to an agent who left or is on vacation must be reassigned" should be declared at the WFMS-level, since it defines a general policy valid for every task. Rules can be ordered by absolute priority. Upon concurrent trigger, the scheduler executes first the rules with higher priority. Among rules with the same priority, the choice is non-deterministic.
145
2.2
Patterns
Since the Chimera-Exc language is quite flexible, the definition of a variety of exceptional behaviors is allowed. However, experience in exceptions design has shown that, although all the language features are required, most triggers follow common, repetitive "patterns". These observations led to the idea of taking advantage of repetitive patterns in triggers definition, in order to reduce the design effort and to provide guidelines for pattern reuse. We introduce patterns as generalized descriptions of triggers and exceptional situations that can frequently arise in WF modeling. Patterns predefine typical rules or set of rules that capture the knowledge about the occurrence of an exceptional situation and the actions that can be performed to deal with it. Patterns consist of predefined parts, parameterized parts, and optional parts. Parameterization and optionality are introduced to support pattern re-usability and adaptation in WF design for the aspects related to exception modeling and handling.
Pattern Description: The pattern are mainly described by a specification part and a sample usage as follows. Specification part: The specification part of the pattern is the description of an exception or of an application, including a set of textual fields and a WPDL and/or Chimera-Exc portion implementing the pattern. In particular, the pattern specification is structured in the following fields: name, uniquely identifying the pattern in the catalog of all patterns; - intent, a textual part describing the purpose of the pattern (intended scope and known uses); - classification, according to the categories and sub-categories of the catalog; - template, containing the core specification of the pattern. It can be provided in two forms: 9 For patterns representing exceptions, the template contains the specification in terms of events, conditions, and actions. On the contrary of the events and conditions, which are the main parts of the patterns, the action part provides only suggestions. This reflect the fact than exception patterns focus on how to capture exceptions, more than on how to fix reactions, which are application dependent: in a given application, a warning message can be a satisfying reaction, when a cancelation of the case can be more suitable in another application of the same pattern. 9 For patterns representing application structures, the template contains the diagrammatic WF schema skeleton describing the application fragment. The template usually contains parametric fields to be filled in with specific values provided by the user; mandatory and optional parts can also be specified. -
146
- keywords, which are a set of user-selected terms that can be used to refer
(select, search, etc.) to the available patterns in the catalog; this field allows one to describe more precisely the topics of the pattern, especially to distinguish the different patterns of a given category in the classification; related to, establishing links among patterns to combine them in different structures; - guidelines, providing suggestions to the user about possible usage and personalization of patterns.
-
In Figure 2, an example of pattern specification is given. It is a pattern to be used to start a new case upon termination of another one.
Pattern Specification Name: Termination Intent: This pattern allows the definition of the initiation of a case upon termination of another case, considering its status (in terms of WF variables).
Guideline: The condition may contain clauses related to values of one or more variables in the terminated case.
Fig. 2. The Termination pattern
The complete description of the pattern-specification language can be found in 8. Sample usage part: Since the user of the catalog is intended to be an expert of the application under design and is not required to have a detailed knowledge of the Chimera-Exc syntax, the pattern model is endowed with the user oriented sample usage pattern. This is a set of instantiations of patterns on specific examples. They show how patterns can be personalized in different contexts and applications by illustrating how parameters of patterns can be supplied by the designer to produce a concrete WF. The sample usage part of the pattern description is a set of WF-specific instantiations of patterns related to an application domain. In general, several examples are provided for a pattern for different domains. The sample usage is an instantiation of the variables/parameters appearing in
147
the template field of the pattern specification according to the domain selected to construct the example. The missingPayraent trigger given in Sec. 2.1 is a sample usage of the pattern t e r m i n a t i o n presented in Figure 2. P a t t e r n R e u s e : Two basic mechanisms are provided for reusing patterns in W F design: specialization and instantiation. Pattern specialization is a mechanism for creating a new pattern starting from an existing one. Pattern instar~tiation is a mechanism for creating triggers for a particular situation by using a pattern.
Pattern specialization: When designing triggers, one may want to specialize a trigger previously defined. Since it was already explained, the action part of the trigger provides only suggestions, on the contrary of the events and conditions parts. Therefore, the specialization focuses on the event and condition parts. Three ways are given to specialize a trigger. They can be combined. The first way consists in rewriting the trigger using more specific genericterms. Consider a pattern composed of a trigger with an event part containing the generic term < r e f e r e n c e E v e n t > . If one wants to write another pattern, with the same structure as the previous one, but focused on external events only, one can specialize the exception part of the pattern by replacing < r e f e r e n c e E v e n t > with the more specialized generic-term < e x t e r n a l E v e n t > . - The second way consists in instantiating part of the trigger. This is done by replacing a generic term by a non-generic one. For example, in a given trigger of a pattern can be specialized into t a s k in a more specific pattern dealing only with tasks. The third way consists in adding conditions and actions to a given trigger in order to obtain a more specialized one. These additions can be done only in the condition and actions parts. Adding events is not allowed since this would change the invocation of the trigger.
-
-
Pattern instantiation: Patterns are used when designing a new W F as follows. After choosing the suitable pattern, this has to be instantiated. In general, this is done by replacing the generic-term with a value, for example < r e f e r e n c e E v e n t > becomes documentArr• A more specific name must also be assigned to the pattern and the triggers. If there are optional parts, the user has to decide if he/she takes the optional parts or not. If several possible actions are defined in the action part, it is also necessary to choose one or several of these actions. It is also allowed to add conditions or actions to the trigger, for example to test the value of a variable which is specific to the W F under design. A more complete and formal description of the instantiation and specialization mechanisms can be found in 8. 3
Specification
of the
Exception
Design
Environment
In this section, we present the environment supporting the pattern-based approach illustrated in Section 2. The tool is mainly given in terms of specification of its functionalities which deal with pattern management in the catalog
148
and with the design and analysis of exceptions. The current prototype environment implementing most of the functionalities for catalog management, and for pattern-based exception design and analysis is finally described.
3.1
Management of Catalog
Management of the pattern catalog deals mainly with the insertion, modification and deletion of patterns. However, it also takes into account the management of the categories useful to classify and retrieve patterns. In this section, the environment functionalities dealing with management of categories are presented, followed by the functionalities for pattern management.
Category management: As presented in Section 2, the catalog is divided in categories ( b u i l t - i n p a t t e r n s , g e n e r i c t a s k s and g e n e r i c - a p p l i c a t i o n o r i e n t e d s t r u c t u r e s ) . Each is divided in sub-categories: Termination, for example is a sub-category of the b u i l t - i n p a t t e r n s category. Since the catalog evolves during time, it is possible to add new categories and sub-categories, and to remove some of them. For removal, a check is performed to ensure that no pattern of the catalog refers to the category or sub-category being deleted. Pattern management: This functionality deals with creation, modification and deletion of patterns. When creating a pattern, its different parts have to been fill-in by the user: name, intent, classification, template, keywords, related-to patterns and guidelines. The pattern name must be unique. The classification conforms to the categories and sub-categories of the catalog. Patterns keywords can be selected from a tool-proposed list of keywords and belong to a thesaurus automatically updated whenever new keywords are specified for newly defined patterns. Guidelines can be textual information to be compiled off-line by the pattern designer. Links between templates ("related-to" fields) can be defined by activating an "add link" and starting a corresponding dialog window. Sample usages can also be written to facilitate pattern reuse. If the specialization mechanism is used to create a new pattern, an existing pattern P can be selected, and the designer can select the parts of P that must be modified. The pattern has to share at least one keyword of its generic pattern (the pattern from which it is derived) and has to be placed in the same category/sub-category. Links are maintained in the catalog between patterns belonging to a specialization chain. When modiying patterns, in addition to the modification of its features (intent, template, keywords, related-to fields and guidelines), new sample usages can be added to an existing template, by defining an "executable example" and by selecting the template of interest from the catalog. It is also possible to change the category and/or sub-category of a pattern. This means changing the location of the pattern in the repository, but also changing the location of the patterns linked to it through specialization or generalization, in order to maintain the catalog consistency.
149
Deletion of patterns is performed by selecting the pattern of interest. The tool then performs a set of basic integrity checks: - if another pattern uses the current one, the link is also removed; - if the pattern is involved in an inheritance hierarchy (is inherited by at least a pattern P2, and possibly inherits also from at least pattern Pl), the user has to choose (i) to delete also P2, or (ii) to build a new inheritance link between Pl and P2.
3.2
Design of Exception
The environment support the pattern-based approach to the design of exceptions in WF schemas limited to the design of its exceptions. In the following, the word exception schema will represent the set of exceptions related to a given WF schema. Exceptions can be designed following a bottom-up process, based as much as possible on reusing the patterns available in the catalog through the instantiation mechanism. The tool guides the designer during the process of exception specification by providing both classification information, which facilitates the search in the catalog, and guidelines to enter values for pattern parameters and to compose patterns into more complex structures. When creating an exception schema, exceptions can be written (created) from retrieved patterns. These patterns are instantiated/personalized for the WF schema at hand. New exceptions can also be defined from scratch, if the available patterns do not fit the WF requirements. Two search modalities are provided, namely, keyword-based and classificationbased search. The first relies on a pre-defined thesaurus of terms loaded at pattern specification time to characterize the pattern within the catalog. Keywords can refer either to pattern elements (e.g., names of events, actions) or can be more generic, domain-related keywords. The second modality allows for the retrieval of patterns based on two orthogonal views of the catalog: - typology of pattern, with respect to event types (e.g., temporal, external, data); - abstraction level of patterns in the catalog (built-in, generic, applicationoriented). Patterns matching search criteria are provided and can be manipulated by the designer. The instantiation consists in binding the pattern parameters on the basis of the semantics of the WF schema, by exploiting the syntax-driven capabilities of the tool and its interface facilities. A set of dialog windows drives the designer in entering the patterns fields, possibly enabling the visualization of the guidelines and usage examples associated with the template under manipulation. New conditions and actions can be added to the exceptions derived from the patterns. The tool dialog facilities support also the definition of new exceptions from scratch. Triggers within a given exception schema can be modified and/or deleted
150
as needed by the designer. No special checks have to be performed, because the triggers, in this part of the tool, are not linked to the patterns of the catalog. Exception schemas can be modified, with respect to their exceptions. These modifications include insertion of triggers, by searching in the catalog or by writing them from scratch, modification of the existing triggers, and deletion of some of them if they are no longer needed. 3.3
Schema Analysis
Exceptions in WFs are typically designed individually, by identifying the exceptional situation, selecting the appropriate pattern from the catalog, and instantiating it in order to obtain a Chimera-Exc exception. However, although an exception considered individually may be correctly specified, mutual interactions among exceptions may lead to unforeseen and undesired behaviors. For instance, a set of exceptions may trigger each other, or two exceptions might operate on the same data, with the consequence that, if they are concurrently triggered, the final result of their execution depends on their non-deterministic execution order. Exceptions analysis aims at statically (i.e., at compile-time) verifying that such undesired interactions do not occur. In particular, it aims at verifying two significant properties for a set of exceptions: termination and confluence 1, 3, 4, 20. Termination holds if exceptions do not trigger each other indefinitely, so that rule processing eventually terminates. Termination analysis is typically based on building graphs depicting triggerings and activations among exceptions, and on detecting cycles in these graphs (see 4, 7 for details). Confluence requires the final effect of the rule processing (represented by direct or indirect changes to the WFMS database) to be independent on the (non-deterministic) scheduling of triggers having the same priority 8. Sufficient conditions for confluence have been proposed in 1, 3, 8. Unfortunately, the above properties are undecidable in the general case. A fully automated analysis of interactions between exceptions is generally difficult to achieve, since it requires to understand the semantics of conditions and actions. A simple, conservative solution consists in checking that exceptions query and operate on different classes. More sophisticated and less conservative techniques for rule analysis, which consider the semantics of conditions and actions, have been proposed in 3. In addition to adapting techniques proposed for active database rule analysis, a tool performing exception analysis may take into account the context in which exceptions are declared; in our model exceptions are always active, and if two exceptions share the same triggering event, they are both triggered as the event occurs, regardless of which part of the case is currently in execution. Exceptions can be concurrently triggered even if they are raised by different events, since exception scheduling occurs periodically, and within two subsequent exception processings many events can occur and many exceptions can be triggered, making the analysis more and more complex. However, although exceptions are always
151
triggerable regardless of the context where they are declared, it is a good design practice to constrain the exception (within the condition part) to be active only when the element in which they are declared (e.g., a task) is running. If this is done, then we can reduce the scope of the analysis to exceptions of tasks that may be concurrently active (besides WF- and WFMS-level exceptions), thus reducing the number of exceptions that need to be concurrently analyzed. This is a key issue in exception analysis, since sufficient conditions for termination and confluence are rarely met when the analysis is extended to the globality of declared exceptions, particularly when conservative approaches are followed.
3.4
Tool A r c h i t e c t u r e
This section presents the architecture and the functionalities provided by the WERDE tool and gives a usage example. The architecture is depicted in Figure 3. The definition of an exception for a WF schema is performed through the Schema Editor module, based on reusing the available patterns. From within the Schema Editor, the designer can interactively search the catalog for a suitable generic pattern using the Exception Retriever module. Retrieved patterns are instantiated/personalized for the WF schema at hand, through the Exception Writer module, which can also be used to define new patterns or exceptions from scratch, if the available ones do not fit the WF requirements. Exceptions defined for a schema can be analyzed by exploiting the Schema Analyser module of WERDE. The Catalog Manager performs the catalog management functions that is, insertion, modification and deletion of patterns. A Visual C + + /Windows prototype of WERDE is available, operating in a Client/Server environment, in accordance with the design guidelines of the other WIDE tools 6.
M a n a g i n g t h e Catalog: The pattern catalog is extensible by defining new patterns and/or specializing the existing ones. To this purpose, WERDE provides maintenance functionalities to manage the catalog (Catalog Manager module) by allowing insertion of new patterns and modification or deletion of existing ones. It also allows to add/remove pattern categories and sub-categories.
Pattern management: Creation: Newly defined patterns are stored in the catalog, under the proper category/sub-category. Fields of the pattern specification are interactively filled in. The pattern template is specified by invoking the Exception Writer editor. Patterns keywords can be selected from a proposed list. Keywords in the thesaurus are automatically updated whenever new keywords are specified for newly defined patterns. Guidelines can be inserted as textual information to be compiled off-line by the pattern designer.
152
T~k-level Task-set level ~ level
| /
SCHEMA ANALYSIS ~
~l
',
/
Schema Analyser
| t
I J
......
I
Creation Modification
i i
'
l
__L ......
.....................
Catalog Manager Category
Pattern
Creation l Modification| Deletion
Addition
Removal .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
CATALOG MANAGEMENT .
.
.
.
.
.
.
.
Fig. 3. Architecture of the WERDEtool
Modification: An existing pattern can be modified, after being selected using the Exception Retriever. The designer can then select the parts of the pattern that must be modified, and operate on them through dialog windows. In WERDE it is possible to add new sample usages to an existing pattern template, by defining an "executable example" and by selecting the template of interest from the catalog. - Deletion: it is performed by selecting the pattern of interest, and the tool performs the basic integrity checks.
-
An example of work session, dealing with the specification of the terminat ion pattern shown in Figure 2, is presented in Figure 4. Insertion/removal of categories: Categories and sub-categories can be added to the catalog at each level. Removal of categories implies the deletion of all subcategories, provided that the designer has already cancelled all the corresponding patterns. U s i n g t h e Catalog: WERDE guides the designer during the process of exception specification by providing both classification information, which facilitates the search in the catalog, and guidelines to enter values for pattern parameters and to compose patterns into more complex structures.
153
Fig. 4. Creation of the termination pattern in WERDE
A typical exception-design session consists in activating the Schema Editor module. The main functionalities that can be invoked from within this module deal with the creation, modification and deletion of exception schemas. The functionalities provided through the creation and modification of exception schemas are the following: - Retrieval, to access the catalog and search for suitable patterns. The two search modalities presented above are implemented (keyword-based and classification-based search). Figure 5a shows the "Retrieve Option" screen where the various search modes are available. In particular, consider that the t e r m i n a t i o n pattern is being searched by keywords, as depicted in Figure 5b. The result of the search appears in Figure 5c. - Insertion, to support the interactive definition of exceptions starting from a retrieved pattern or from scratch. The instantiation consists in binding the pattern parameters on the basis of the semantics of the WF schema, by exploiting the syntax-driven capabilities of the Exception Writer module and the template interface facility of W E R D E . This facility consists of a set
154
Fig. 5. Pattern retrieval. Selection of retrieval option (a), keyword specification for keyword-based retrieval (b), and presentation of the retrieved pattern (c)
155
of dialog windows which drives the designer in entering the patterns fields, possibly enabling the visualization of the guidelines and usage examples associated with the template under manipulation. The dialog facilities of the E x c e p t i o n W r i t e r support also the definition of new exceptions from scratch. The instantiation of the t e r m i n a t i o n pattern retrieved through the steps depicted in Figure 5 works as in the sample work session of Figure 6, where the pattern is instantiated to obtain the trigger missingPayment of section 2.1. The instantiation steps consist of activating the exception writer module, selecting the item to be instantiated (<wfName>W in the example of Figure 6), and overriding it with the desired Chimera-Exc condition
(roomReservation (R)).
Fig. 6. Instantiation of the termination pattern
-
Modification, to modify exceptions inside exception schemas.
-
Deletion, to remove exceptions from exception schemas.
-
A n a l y z e , to analyze the exceptions written for the current schema, using the S c h e m a A n a l y s e r module. W E R D E currently provides a basic set of the static analysis functionalities, by focusing on interactions within each pair
156
of exceptions. In WERDE, two triggers are considered as conflicting if they are active simultaneously and contain incompatible actions; actions are incompatible if the order of their execution leads to different states of the WF management database 6. A table of incompatible actions is maintained to support the analysis, which can be performed at three levels: 9 task level, to check exceptions defined for a given task; 9 task set level, to select a number of tasks for which the exceptions must be analyzed; 9 schema level, to check all the exceptions defined in the schema. Anomalous situations detected by the Analyser are signaled to the designer who can activate the Exception Writer for editing the involved exceptions. 4
Concluding
Remarks
We have presented the WERDE environment supporting the design of exceptions in WF schemas. The environment is based on a catalog of WF patterns which are a generalized description of triggers which can be reused when designing a new WF schema. Exceptions model both abnormal situations and typical start/end/repetitions of operations in a WF. We have presented the structure of the pattern catalog and the mechanisms for pattern reuse. We have also described the related functionalities of the WERDE tools for exception design and catalog management. Currently, the catalog is populated with a set of patterns which have been constructed by analyzing a set of real cases provided in the WIDE project testbed. A set of primitives for pattern reuse has been fully specified in 8 and is being tested to check the quality of these patterns in terms of significance and accessibility for reuse. Particularly the sample usages of patterns stored in the catalog are proving to be an effective means for guiding the designer in selecting the appropriate patterns, in instantiating them appropriately, and in inserting the exceptions correctly in the WF schema. The implementation of W E R D E has currently completed most of the functionalities described in the paper. Some functionalities, such as pattern design and schema analysis, are being extended in order to achieve a more complete and effective suite of functions. For example, for schema analysis, we plan to consider exception termination analysis and confluence analysis on rule sets, and to consider in more detail the semantics of actions and conditions. For pattern design, we plan to have functionalities which help the catalog administrator in abstracting exception skeletons from WF schemas in a guided way and storing new patterns in the appropriate categories with suitable links to the existing repository. In fact, the catalog is regarded as an extensible tool to be populated gradually with new categories of patterns to be defined by examining further real WF applications. Further points of research regard the use of exceptions as a basis for WF dynamic evolution and the design of a user feedback mechanism for validation of the quality and effectiveness of a given WF, using adaptive techniques such as those studied in 11.
157
Acknowledgments: The authors are thankful to B. Pernici and S. Castano for common work and ideas and to the students who implemented the tool.
References 1. A.Aiken, J.Widom, J.Hellerstein. "Behavior of database production rules: termination, confluence, and observable determinism". Proceedings of SIGMOD, San Diego, CA, 1992. 2. S. Bandinelli, A. Fuggetta, C. Ghezzi, "Software process model evolution in the SPADE environment", IEEE Transactions on Software Engineering, December 1993. 3. E. Baralis, J. Widom, "An algebraic approach to rule analysis in expert database systems", Proceedings of VLDB 94. 4. E.Baralis, S.Ceri, S.Paraboschi. "Improving rule analysis by means of triggering and activation graphs", Proceedings of the 2nd International Workshop of Rules in Database Systems, Athens, Greece, 1995. 5. R. Bellinzona, M.G. Fugini, B. Pernici, "Reusing specifications in OO applications", IEEE Software, vol. 12, n. 2, March 1995. 6. F. Casati, P. Grefen, B. Pernici, G. Pozzi, G. Sanchez, "WIDE workflow model and architecture", Technical Report, Politecnico di Milano, n. 96-050, 1996. 7. F. Casati, S. Ceri, B. Pernici, G. Pozzi, "Specification of the rule language and active engine of FORO v . l ' , WIDE Technical Report n. 3008-6, September 1997. 8. F. Casati, S. Castano, M.G. Fugini, I. Mirbel, B. Pernici. "Using patterns to design rules in workflows", Technical Report n. 97-065 of the Politecnico di Milano, November 1997. 9. S. Castano, M.G. Fugini, I. Mirbel, B. Pernici, "Workflow reference models", WIDE Technical Report n. 3015-2, June 1997. 10. S. Ceri, R. Ramakrishnan, "Rules in database systems", ACM Computing Surveys, Vol. 28, No. 1, March 1996. 11. E.Damiani, M.G. Fugini, E. Fusaschi, " COOR: a Descriptor Based Approach to Object-Oriented Code Reuse", IEEE Computer Magazine, September 1997. 12. U. Dayal, M. Hsu, R. Ladin, "Organizing long-running activities with triggers and transactions", Proceedings of ACM SIGMOD, 1990. 13. D. Georgakopoulos, M. Hornick, A. Sheth, "An overview of workflow management: from process modeling to workflow automation infrastructure, distributed and parallel databases", Vol. 3, n. 2, April 1995. 14. C. Ghezzi, M. Jazayeri, D. Mandrioli, "Fundamentals of software engineering", Prentice-Hall Intl., 1991. 15. D. Hollingsworth, "Workflow management coalition, the workflow reference model", Technical Report n. TC00-1003, November 1994. 16. R.E. Johnson, "Frameworks = components + patterns", in Communications of the ACM, vol. 40, n. 10, October 1997 17. G. Kappel, P. Lang, S. Ransch-Schott, W. Retschitzegger, "Workflow management based on objects, rules, and roles", IEEE Data Engineering Bulletin. Special issue on WF Systems, vol. 18, no. 1, March 1995. 18. C.W. Krueger, "Software reuse", ACM Computing Surveys, vol.24, n.2, June 1992. 19. D.C. Rine, "Supporting reuse with object technology", IEEE Computer, Special Issue on OO Development and Reuse, October 1997. 20. J. Widom, S. Ceri, "Active database systems: triggers and rules for advanced data processing", Morgan Kaufmann, San Mateo, CA, 1996.
A u t o m a t i n g Handover in D y n a m i c Workflow Environments * Chengfei Liu 1.* and Maria E Orlowska 2 and Hui Li 2 1 School of Computing Sciences, University of Technology, Sydney, NSW 2007, Australia Department of Computer Science and Electrical Engineering, The University of Queensland, Qld 4072, Australia
A b s t r a c t . Workflow technology has been widely used in business process modelling, automation and reengineering. In order to meet the fastchanging business requirements, to remain competitive in the market, an enterprise may constantly refine the workflow models of its business processes. The most challenging issue in evolution of a workflow model is the handover of its running instances from the old specification to the new specification. Such a handover depends on the semantics of a workflow model as well as the execution information of its running instances. A handover policy, therefore, needs to be specified for this purpose. In this paper, we propose a simple yet effective handover policy specification language. Using this language, a designer can easily specify a handover policy which reflect exactly what a workflow administrator needs to react to when a workflow model evolves. Criteria for the correct specification of handover policies are also addressed. Finally, a framework for automating handover of workfiow instances is presented.
1
Introduction
Recent years have seen widespread use of workflow technology in business process modelling, automation and reengineering. Next only to the Internet related technology and products, the workflow technology and products 5, 11 are arguably the most influential new breed of software systems from the perspective of achieving significant impact on enterprises. Many enterprises have shifted their data-centric approach in the context of the information systems technology and solutions to a process-centric one. Work flow technology has matured to some extent, and current products are able to support a range of applications. However, m a n y limitations remain in current workflow technology, especially for supporting more demanding applications and more dynamic environment. The work reported in this paper has been funded in part by the Cooperative Research Centres Program through the Department of the Prime Minister and Cabinet of the Commonwealth Government of Australia. *~ Work done partially while the author was at the Distributed Systems Centre, Bris w bane, Australia
160
In a fast-changing environment, an enterprise may constantly refine its workflow models to remain competitive in the market, to meet customers' new requirements, to change business strategies, to improve performance and quality of services, to benefit from changes in technology, etc. Two aspects are related to the evolution of a workflow model. First, the old specification of a workflow model needs to be changed to a new specification correctly. This is the static aspect of the evolution. Second, as a business process tends to be long lasting, whenever a workflow model changes its specification, there may exist a set of running instances of the old specification. How to handover these running instances is an interesting and challenging issue. Whether or not the running instances shall evolve according to the new specification and how they evolve depend on an evolution policy which is specific to the business process which the workftow model represents for. We call this evolution policy as a h a n d o v e r policy. Currently, workflow evolution has not been sufficiently supported by workflow products. Only very primitive policies are supported by some workflow products such as F o r t e C o n d u c t o r 4. Other workflow products such as I n C o n c e r t 6 support dynamic workflow adaptation at the instance level 14. Schema evolution has been widely addressed in the field of Object-Oriented Databases and Software Processes 1, 16, 7. However, little work has been done in addressing the problem of workflow evolution, particularly the dynamic aspect. In their paper 3 Casati et al. have presented a workflow modification language that supports modification of a workflow model (schema). They have also discussed the case evolution policies and have devised three main policies to manage case evolution: a b o r t - to abort all running instances and to start new created instances following new specifications, f l u s h - to finish all running instances first and then to allow new instances to start following new specifications, and p r o g r e s s i v e - to allow different instances to take different decisions. Though the progressive policy is further discussed in their paper, the fine granularity of the case evolution policy specitications has not been addressed. We view a workflow model evolution as a process which consists of three steps: (1). to modify a workflow model from its old specification to its new specification. (2). to specify a handover policy for handing over the running instances of the workflow model to be evolved. (3). to apply the handover policy. A workflow model modification can be done by applying a series of modification primitives using a workflow model modification language. In our study, we focus on the handover policy which is formulated based on the old and new specifications of a workflow model. When specifying a handover policy, we assume that a specifier has knowledge of both old and new specifications of the workflow model as well as their difference. Step 1 and Step 2 are performed at build-time, while only Step 3 is performed at run-time. The rest of the paper is organized as follows. In Section 2, we brief specification of workfiow models. In Section 3, we design a handover specification language for specifying what a workflow administrator needs to react to the running instances when a workflow evolution occurs. Correct specification of handover policies is addressed in Section 4. In facilitating handover of workflow
161
instances, a framework for implementing the handover specification language is presented in Section 5. Section 6 concludes the paper with indication of our future work.
2
Workflow Model Specification
As the specification of a handover policy for an evolution of a workflow model is based on the old and new specifications of the workfiow model, we first review the workflow modelling work. Several workflow modelling techniques have been proposed in the literature 12,2, 8, 13, some of them even target the transactional aspects of workflows. In 13, a graphical workflow specification language is proposed for workflow conceptual modelling. As shown in Figure 1, the language includes four types of modeling objects: task, condition, synchronizer, and flow. The flows are used to link the first three types of objects to build the specification of a workflow model.
Task
9
V
Condition
Synchronizer
Flow
Fig. 1. Modelling Objects of Workflows
- Task - A task is a logical step or description of a piece of work that contributes towards the accomplishment of a workflow model. It can represent both automated activities and human activities. Tasks are performed by assigned processing entities. A workflow specification is basically used to specify the coordination requirements among tasks. Sometimes properties of tasks may also be specified in capturing more aspects (e.g., transactional aspects) of a workflow model. However, the actual semantics of tasks is beyond the scope of workfiow specifications. - Condition - A condition is used to represent alternative paths in a workflow specification depending on a conditional value. - Synchronizer - At certain points in workflows, it is essential to wait for the completion of more than one execution path to proceed further. A synchronizer is used for this purpose. - Flow - A flow defines the connection between any two objects, other than flows, in the workflow specification. It shows the flow of control or data from one object to another. A workfiow model can be represented as a workflowgraph using these modelling objects, where nodes of the graph can be tasks, conditions and synchronizers and links of the graph are flows. Restriction is placed in constructing a correct
162
workflow graph. Only a limited yet relatively complete set of constructs are supported by the language. They are Sequential, Exclusive OR-Split (Alternative),
Exclusive OR-Join, AND-Split (Parallel), AND-Join (Synchronization), Nesting, Iteration, Start/Stop. Besides, a set of correctness constraints of workflow graphs have also been identified and verification algorithms have been proposed for verifying the syntactical correctness of workflow graphs specified using this language 15,13. For instance, all or none of the flows proceeding a synchronizer activate for all possible instances of a workflow. Only one or none of the flows leading to a task or a condition activates for all possible instances of a workfiow. The first rule eliminates the possibility of a synchronizer deadlock. The second rule eliminates the possibility of an unintentional multiple execution. In this study, we use this graphical language to specify workfiow models and assume that workflow graphs of both the old and new specifications of workflow models specified using this language are syntactically correct. As handover is difficult to make inside an iteration block, we treat an iteration block as a single task. 3
Handover
Policy
Specification
A handover policy is specified to handover current running instances of a workflow model which is to be changed. It is used to model the dynamic aspect of workflow evolution. In this section, we design a handover specification language. The objectives of the language is effective yet simple. As when a handover policy is applied to an evolution of a workfiow model (i.e., from its old specification to its new specification), the running instances may be executing at any task of the old specification. W h a t is worse, different instances can take different paths to the same task. Therefore, different instances may require different handover strategies. No matter how complex the situation can be, the language should be expressive enough for specifying all a workflow administrator wants to specify. Obviously, if all the possibilities need to be specified explicitly, it can be cumbersome, even not applicable to large workflow models. Therefore, simplification of specification must be considered. Fortunately, in practice, a workflow administrator is only interested in some key points where turning actions need to be taken. Using some default and grouping specification, the specification of a handover policy can be greatly simplified. In the following, we discuss the handover specification language.
3.1
Syntax of the Language
Associated with every workflow model evolution is one and only one handover policy. A handover policy is defned by a set of handover statements. Three handover aspects of a running instance are described in each handover statement: - current position - indicating current executing task of a running instance; - history - indicating the traversed paths of a running instance by conditional value;
163
-
action - indicating the action to be taken. A BNF definition of the handover policy specification language is given below:
::= {} ::= <do clause> ::= ON <position specification> <do clause> ::= DO I IF DO ELSE
<do clause>
P o s i t i o n s p e c i f i c a t i o n A position of a running instance is specified by the current executing task of that instance. In general, the exact point that the scheduler can interact is the completion point of the executing task of an instance. For the purpose of simplified specification, multiple tasks can be grouped to share a common do clause. Two ways of grouping are used: tasks without order and tasks with order. Position specification is further defined as follows: <position specification>
::= I "{" {,}"}" I TO
Action specification In supporting handover of a running instance, two important actions must be supported. One action is rollback. It is used to semantically undo some work so that the running instance can comply with the new workflow specification. A destination task must be given for a rollback action. The other is change-over. It is used to migrate the execution of a running instance (or a path of it) to follow the new specification. A destination task may or may not be given for a change-over action. If a destination task is not specified in the change-over action, the task which has the same name as in the current (old) specification in the new specification is chosen as the default destination task. There is another action called go-ahead which may not be explicitly specified. If no handover statement is defined on a task, the default handover action after the execution of the task is going ahead. As such, the specification of a handover policy can be greatly simplified. Only the turning points need to be specified. This coincides how a workflow administrator behaves to cope with a handover. Action specification is defined as follows:
::= ROLLBACK T0 i CHANGE 0VER T0 i G0 AHEAD
Conditional t u r n i n g s Sometimes, a turning action at the current task is decided based on the history (i.e., the traversed paths) of a running instance. This is supported by the conditional turning by representing the history information in a conditional wlue. Besides the history information, other semantic information (e.g., time) can also be specified in the condition to facilitate flexible handover.
164
3.2
Handover Policy Examples
We use the PC assembling workfiow example introduced in 3 to illustrate how a handover policy can be specified using our handover specification language. As shown in Figure 2, the old assembly process starts by preparing in parallel a cabinet (either a Tower or a Minitower) and a motherboard (including CPU and disk controller). Then the motherboard is inserted into the cabinet. After the FDD is inserted, a condition on Container is checked to determine whether a cdrom needs to be inserted. Finally, the hard disk and video ram are inserted. The assembly process changes with the requirement that the cd-rom is replaced by NiceLab's cd-rom 4x and audio card. Based on the different decisions, different handover policies can be specified as follows.
-S (~
Ccnt~ner ffiTower
\/
Container = Tower ~ t a i n e r . Mlnltower
Old specification
New specification
Fig. 2. Old and New Specifications of PC Assembling Workflow Model
165
Example 1. Specification of concurrent to completion policy, i.e., all running instances are allowed to terminate following the old specification, while new instances can be started following the new specification. This policy can be expressed by the following single statement. ON Start TO Plug-Video-Ram DO GO AHEAD.
Example 2. Specification of Abort policy, i.e., all running instances are aborted, and the newly created instances will start following the new specification. ON Start TO Plug-Video-Ram DO ROLLBACK TO Start.
Example 3. Change over the running instances if executing before the condition checking on Container, otherwise go ahead. ON Insert-FDD-l.44M DO CHANGE-OVER.
Example 4. Change over before the condition checking on Container if the instance will satisfy the condition Container -= Tower; Rollback to the end of the task Insert-FDD-l.~M if the instance takes the path affected by the change. ON Insert-FDD-l.44M IF Container = Tower DO CHANGE-OVER; ON Insert-CD-Rom DO ROLLBACK TO Insert-FDD-l.44M; ON ~Add-I.6GB-HD, Plug-Video-Ram) IF Container = Tower DO ROLLBACK TO Insert-FDD-1.44M.
As shown by above examples, handover policies can be easily and directly specified using the handover specification language. A specifier only needs to explicitly specify the turning actions taken at turning points. In Example 4, a handover policy which consists of three explicit handover statements is specified. The instances executing along the path which is not affected by the change take default action, i.e., go-ahead. This example shows that arbitrary handover policies can be specified using the handover specification language. 4
Correctness
Issue of Handover
Policy Specification
With a handover specification language, workflow specifiers have the flexibility to support fine-granularity of handover policies. However, it may also bring the errors into specifications. As the correctness checking of a workflow model specification, it is also important to check whether a handover policy is specified correctly. As the specification of a handover policy is different from the specification of a workflow model, new correctness problems may exist for a handover policy specification. In order to study the correctness issues of handover policy specification, we first introduce a so-called handover graph. Each handover policy can be defined by a handover graph. The handover graph is constructed based on the workflow graphs of both old and new specifications. Each handover statement is reflected
166
in the handover graph as follows: (1). If a rollback action is defined on a task T, add a link from T to the task to which it rolls back; add a special dead task Td and move all links originated from T to Td. A dead task is a task which never gets executed. (2). If a change-over action is defined on a task T, add a link from T to the task to which it changes over; add a dead task Td and move all links originated from
T to Td. (3). If a (default) go-ahead action is defined on a task T, keep the workflow graph of the old specification unchanged for T. (4). If a conditional turning is specified, add a condition task Tc with two links originated from Tc specifying the two exclusive condition values. Depending on the turning action change the graph accordingly. Example 5. The handover graph for the policy defined in Example 4 is given in Figure 3. 4.1
Syntactical
Error Types
As the handover graph for a workfiow model evolution is constructed based on workflow graphs of the old and new specifications of the workflow model and these workfiow graphs are assumed syntactically correct, a handover graph can be erroneous only if the turning actions are specified incorrectly. Therefore, we aim at errors resulted from incorrect specification of turning actions. The error types which we have identified in specifying a handover policy include: - cyclicness
If a cycle appears in the handover graph of a handover policy, the running instances in the cycle will execute endlessly. Such a cycle must be avoided during specification of a handover policy. In Example 4, if we change the condition of either the first or the third handover statement to Container = Minitower, a cycle will occur and the execution will never stop. - deadlock For a synchronizer node of the handover graph of a handover policy, if a dead task appears in one incoming path but does not appear in another path, a deadlock problem exists for the handover policy specification. For example, if we change over one branch of a parallel construct while keeping another branch go ahead, a deadlock will occur. Another example of deadlock can be resulted from structure mismatch. If we want to change over branches of an alternative construct to a parallel construct, a deadlock will happen to the new specification. - unintentional multiple execution Similar to a workflow model specification, an unintentional multiple execution error may occur in a handover policy specification. One such example may come from changing over multiple parallel branches directly or indirectly to the same task of the new specification. Another example can be changing over different branches of a parallel construct to different branches of an alternative construct, respectively.
167
\
r |, ~ qCo~r =Tower
Comaf~er=Towe~
~'--~ Fig. 3. A Handover Graph
4.2
C o r r e c t n e s s C r i t e r i a for H a n d o v e r P o l i c i e s
In preventing handover policy specification errors, we define a set of correctness constraints as follows.
Rule 6. At most one handover action may be executed for each task of a running instance. In other words, if a cycle appears in the handover graph of a handover policy, then the conjunction of conditions specified in all condition nodes along the cycle must be false, i.e., the cycle is a pseudo cycle and no real cycle is allowed in the handover graph. Rule 7. Either all branches or no branches of a parallel construct of the old specification are changed over to new specification. In other words, either all dead tasks or no dead tasks can be connected to a sychronizer.
168
Rule 8. Branches of an alternative construct in the old specification cannot be changed over to branches of a parallel construct in the new specification.
Rule 9. Branches of a parallel construct in the old specification cannot be changed over to branches of an alternative construct in the new specification. Rule 10. Branches of a parallel construct in the old specification cannot be changed over directly or indirectly to the same task of the new specification. These rules are verified for each handover policy specification before it is applied to running instances, thus run-time handover errors can be greatly reduced.
5
Facilitating Handover Policies
As automatic handover of running workflow instances has not been addressed by existing workflow management systems, it is ideal to put forward a framework which can facilitate handover based on current workflow technology. In this section, we address some key technical points towards such a framework.
5.1
Required Data Structures
The data structure for workflow instances is designed as follows:
WFInst(InstID, State, History, ActivePath(SpeeID, CurrentPosition)) Where InstID is used for identifying a workflow instance. State records the current state of the workflow instance, such as, executing, completed. Two additional states migrating and migrated are introduced for handover purpose. The migrating state indicates that the workfiow instance is under a handover process, when the handover process is finished, the state of the workflow instance is changed to the migrated state (not turning back to the executing state). The migrated state indicates that the instance needs to be treated specially in case rollback or another handover (due to newer specification or version of its workflow model) may be required to the instance later since it has undergone a handover process. History records the log data of all traversed paths of the workflow instance. A workflow instance may contain several active parallel paths. The number of paths will increase after an AND-Split is reached and will decrease after a synchronizer (AND-Join) is reached. Every active path has a SpecID and a CurrentPosition associated with it. A SpecID is used for identifying the specification on which the execution of that active path is based. During handover, it is possible that one active path is running on the old specification while another is running on the new specification. A CurrentPosition records the currently executing task of that path. In addition, a new data structure for policy specification is designed.
Policy( Specl D , NewSpecl D, Turning(Task, Condition, Action, Destination))
169
Where SpeclD and NewSpeclD are used for identifying the old and new workfiow specifications on which a handover policy specification is based. As one and only one policy is associated with each workflow model evolution, a policy can be identified by a SpeclD. Several turnings can be described in a policy. Each Turning records a task - after its completion the turning will take place, a condition - the turning may take effect only under the condition, an action either rollback or change-over, and a destination - indicating the destination task of the turning. 5.2
Applying a ttandover Policy
To apply a handover policy to a workflow specification, a workflow system command can be issued: handover( Specl D ) This command will automatically change the state of all running instances of the specification indicated by SpeclD to the state Migrating. 5.3
Scheduling a Handover A c t i o n
When a task t indicated by CurrentPosition in an active path p of a running workflow instance w finishes its execution, the scheduler will schedule the next step for executing. Two cases are scheduled differently: (1) If the instance w is in a state other than Migrating, the scheduler will schedule w (specifically the path p) as usual, i.e., to take the next step from t according to the specification of p indicated by SpeclD. The scheduling information is recorded in the History. (2) If the state of the instance w is Migrating, the scheduler will first find the policy defined on the specification of p indicated by SpeclD. After that, it tries to match t with the Task in a Turning of the policy, if it matches one and the Condition in the Turning is satisfied, then take the Action in the Turning. Otherwise keep trying. If there is no one matches, take the default next step as usual according to the specification of p indicated by SpeclD. T a k i n g a Rollback Action If a rollback action is scheduled, the path p of the workflow instance w is rolled back to the point specified by Destination. The rollback process may be undertaken with the help of partial compensation 8, 10, 9 which is another technical issue in workflow management systems. The information of the rollback action and all the rollback steps need to be recorded in the History. After the rollback action is completed, CurrentPosition is changed to hold the Destination. T a k i n g a Change-Over Action If a change-over action is scheduled, the path p of the workfiow instance w is changed to run on the new specification. The point where the path continues in new specification is specified by Destination.
170
The information of w needs to be modified as follows: (a) SpecID for the path p is changed to hold the new specification ID, i.e., NewSpecID. (b) CurrentPosition for the path p is changed to hold the Destination. (c) A log item indicating the change-over action needs to be added to the History. (d) If all active paths of the workflow instance w are changed over to running on the new specification, the state of w is changed to Migrated and the state change is also recorded in the History.
6
Conclusion
and Future
Work
It is an important yet challenging topic to handover running workflow instances for enterprise computing. In this paper, we specifically addressed this topic. To support flexibility and fine-granularity of handover policy specification, we designed a simple yet effective handover policy specification language. Using this language, a designer can easily and directly specify a handover policy which reflect exactly what a workflow administrator needs to react to when a workflow model evolves. Correct specification of handover policies was also discussed. A framework for automating handover of running workflow instances was presented. In the future, we will investigate algorithms for verifying the correctness of handover policy specifications. As well we will extend our current two-version support of workflow models (i.e., old specification and new specification) to multi-version support (therefore multiple handovers of long-running instances).
References 1. J. Banerjee, W. Kim, H-J. Kim, and H. Korth. Semantics and implementation of schema evolution in object-oriented databases. In Proceedings of the ACM SIGMOD International Conference on Management o/Data, pages 311-322, 1987. 2. F. Casati, S. Ceri, B. Pernici, and G. Pozzi. Conceptual modeling of workflows. In Proceedings of O0-ER conference, pages 341-354. Lecture Notes in Computer Science, Vol. 1021, Springer, 1995. 3. F. Casati, S. Ceri, B. Pernici, and G. Pozzi. Workflow evolution. In Proceedings of the 15th ER Int. Conf., pages 438-455. Lecture Notes in Computer Science, Vol. 1157, Springer, 1996. 4. Forte. Forte Conductor Process Development Guide. Forte Software, Inc., 1994. 5. Butler Group. Workflow: Integrating the Enterprise, June 1996. 6. InConcert. InConcert Technical Product Overview. InConcert Inc., January 1997. 7. M. Jaccheri and R. Conradi. Techniques for process model evolution in EPOS. IEEE Transactions on Software Engineering, 19(12):1145-1156, December 1993. 8. D. Kuo, M. Lawley, C. Liu, and M. Orlowska. A general model for nested transactional workflows. In Proceedings of the International Workshop on Advanced Transaction Models and Architectures, pages 18-35, 1996. 9. F. Leymann. Supporting business transactions via partial backward recovery in workflow management systems. In Proceedings of BTW'95, pages 51-70, 1995.
171
10. C. Liu and M. Orlowska. Confirmation: Increasing resource availability for transactional workflows. Technical report, Distributed Systems Technology Centre, September 1997. 11. Workflow Management Coalition Members. Glossary - A Workflow Management Coalition Specification. Workflow Management Coalition, November 1994. Softcopy available via: http://www.aiai.ed.ac.uk/project/wfmc/. 12. M. Rusinkiewicz and A. Sheth. Specification and execution of transactional workflows. In W. Kim, editor, Modern Database Systems: The Object Model, Interoperability, and Beyond. Addison-Wesley, 1994. 13. W. Sadiq and M. E. Orlowska. On correctness issues in conceptual modeling of workflows. In Proceedings of the 5th European Conference on Information System, 1997. 14. A. Sheth and K. Kochut. Workflow applications to research agenda: Scalable and dynamic workflow coordination and collabration systems. In Proceedings of the NATO ASI on Workflow Management Systems and Interoperability, August 1997. 15. A.H.M. ter Hofstede, Maria E. Orlowska, and J. Rajapakse. Verification problems in conceptual workflow specification. In Proceedings of the 15th ER Int. Conf., pages 73-88. Lecture Notes in Computer Science, Vol. 1157, Springer, 1996. 16. R. Zicari. A framework for schema updates in an object-oriented database systems. In Proceeding of 7th International Conference on Data Engineering, pages 2-13, 1991.
Document-Centric Groupware for Distributed Governmental Agencies Daniel A. Tietze*, Ajit Bapat*, Rolf Reinema** GMD - German National Research Center for Information Technology * Integrated Publication and Information Systems Institute (IPSI) **Institute for Telecooperation Technology (TKT) {daniel.tietze I ajit.bapat I rolf.reinema} @grad.de
Abstract The distribution of the German government between Bonn and Berlin calls for the technical support for the collaborative document-based tasks performed by inter- and intra-ministerial work groups. The currently available cooperation and communication tools do not provide an integrated and homogeneous cooperation environment suitable for this application domain. This paper presents the Cooperative Document Repository, an integrated document-centric collaboration environment. The integration of cooperative document management and document processing with multi-point audio/video conferencing provides a flexible collaboration environment within which a wide range of collaborative activities can be performed.
Keywords: groupware; CSCW; document-based collaboration; federal government; distributed administrations; POLIWORK; video conferencing
Introduction The German unification and the resulting move of governmental agencies from Bonn to the new capital, Berlin, will result in a distribution of administrative units between Bonn and Berlin. This distribution will be both along vertical as well as horizontal lines, meaning that some ministries will be moved entirely to Berlin, some will remain in Bonn, while others will be distributed between Bonn and Berlin. As a result, work groups and task forces (both within a ministry as well as between ministries), which were previously co-located will be distributed as well and will require technical support in order to continue their work. To address the challenges of this distribution and to provide technical solutions and organizational suggestions for the resulting distributed work processes, the German government instated the POLIKOM research program in 1994, within which four projects are addressing the different issues arising from the close cooperation between distributed governmental agencies.
174
At GMD, the authors are involved in one of these projects: POLIWORK z Telecooperation and cooperative document management at the personal workplace which began in 1994 and will continue until the end of 1998. The focus of this project is to support the cooperative tasks of the governmental agencies, specifically those collaborative activities centering on document management and document processing, both in synchronous as well as in asynchronous settings, through the development and installation of appropriate support technology. The user community of the project consists of two German federal ministries, the Ministry of the Interior (BMI) and the Ministry of Economics (BMWi). This paper describes the central software component developed for the user community: the Cooperative Document Repository (CDR), an integrated collaboration support system which inter- and intra-ministerial work groups can use in order to structure and store the documents involved in their cooperative work processes as well as to initiate the required communication and collaboration tasks. The remainder of the paper is organized as follows. The next section presents brief scenarios of collaborative activities found in the user community. From these a set of requirements is derived to be addressed in the subsequent system design. The architecture and functionality of the CDR are then presented, followed by a description of the approach of document-centric collaboration support. The paper concludes with a comparison to related work in the field of collaboration support technology and a brief outlook on future work.
System Requirements The requirements for the technical solution which are presented in this section were gathered in the user community through an analysis of the cooperative work processes found as well as through interviews with the users. The resulting work process scenarios were analyzed in order to determine how to adequately support the arising tasks by introducing a collaboration support system combining state-of the-art communication hardware with advanced groupware technology. Also, the existing infrastructure found within the ministries was examined in order to determine in which way it could be integrated into a collaboration support environment.
Collaborative Activities In the scope of the project, we concentrated on two main scenarios for cooperative activities in the ministerial environment: joint text creation by a work group as well as presentation and discussion including the higher levels of the ministerial hierarchy. A typical task in the application field is the preparation of a legislative draft. Usually, such a text is created jointly by a small group of people. Different parts of the document are created by different authors and then integrated into one common z The work presented in this paper is partially funded by the German Federal Ministry of Education, Science, Research and Technology (grant number 01 IT 403 A-D).
175
document. The various intermediate versions of the draft are passed around among the members of the work group. Paper documents are distributed using fax or regular mail while documents in electronic form are sent via e-mail. To coordinate the integration of the various parts bilateral communication (e.g. telephone call) as well as multilateral communication (e.g. meeting) takes place. Meetings include a certain amount of preparation (setting a date, distributing the necessary documents, etc.) as well as wrapping up (writing minutes etc.). Another scenario is closely related to the one described above. At the end of the joint text creation the resulting draft needs to be approved by the higher levels in the ministerial hierarchy. For this, the final draft is passed to the higher level and then, in a meeting with the superior the draft is presented and discussed, changes to be made are annotated. Afterwards, the document is revised accordingly.
Requirements Based on the work processes found in the ministerial context, a number of requirements can be identified for a system that is to support the distributed, cooperative tasks in the application field. (R1) All work processes are document-based. Either texts are created and edited as part of the process or the process involves the presentation, discussion and annotation of documents. Therefore, document-based support is indispensable. This requirement combines several important aspects: Documents need to be integrated directly into the work processes, including the support of existing document bases (RI.1). In order to work on the documents, cooperative document processing is required. One important aspect of this is the support of legacy applications (R1.2), since the ministries have a certain set of document processing tools which they regularly use (e.g. one of the established office suites). Documents have to be passed to other people involved in the task. Thus, document exchange is to be supported (R1.3). Furthermore, storage of the documents is necessary (R1.4). And finally, paper-based documents need to be integrated (R1.5). (R2) The second important aspect is the provision of communication channels between the people involved in the work processes. Synchronous communication (R2.1) and the high quality of the communication channels (R2.2) are the major requirements. Since the collaborative processes span multiple sites, the communication support should not only be bilateral but also support multi-party distributed conferencing (R2.3). (R3) One key aspect with respect to the acceptance of the system by the users is the usability of the supporting system. To achieve this, several aspects have to be considered: A homogeneous user interface (R3.1), integrated access to various functionalities of the system (R3.2), as well as a familiar look and feel of the system's user interface (R3.3). (R4) Due to the fact that the system is to be used in the Federal Ministries the existing technical infrastructure found there needs to be taken into account. The ministries are connected by an ISDN network which is physically separate from the
176
public telephone and ISDN networks and therefore regarded as being secure. Thus, communication should run over these lines (R4.1). In addition, the government is setting up a governmental IP backbone that connects all ministries within and among the two cities of Bonn and Berlin. Again, this network is to be used (R4.2). Additionally, the system that is to be developed should run on the governmental intranet which is currently being installed (R4.3). (R5) In addition to the previous requirements, some general requirements result from the fact that the system is developed in a pilot project. Adaptability to other user fields, e.g. other ministries (R5.1), as well as scalability to larger numbers of users (R5.2) are required.
System Design In order to ensure rapid availability of the key functionalities required in this project a major design goal was to take already existing components from the market. By doing so, an initial set of requirements could be fulfilled by off-the-shelf components. These application modules then needed to be integrated into the system in a way which makes the combined functionality of the tools easily available to the users. A first version of a collaboration support environment has already been installed at an early stage so that the users could become familiar with the new possibilities offered by communication and cooperation technology. Support for joint viewing and editing of documents by two or more participants is provided by the use of a standardized application sharing package which enables several users at the same time to jointly view and interact with any application and thus any kind of document. In this way, the standard application suites and legacy applications which were already used within the ministries as well as the documents produced by these are integrated into the collaborative processes in a familiar and easy to use manner. In addition to this the use of a color flatbed scanner made it possible for the users to import paper based documents and enabled them to work on them. Both the sharing of any application between several sites as well as the possibility to import any paper-based document fulfilled the requirement for joint viewing and editing of existing document bases. As already mentioned above, the users have asked for high quality synchronous audio/visual communication while working together on documents. This resulted in the provision of off-the-shelf components for audio/video and data conferencing. The two user communities within the project, the German Federal Ministry of Economics (BMWi) and the Federal Ministry of the Interior (BMI) have had different needs for communication and collaboration support 1. While the BMWi has expressed the need for desktop-based collaboration between single users in the form of desktop workstations equipped with conferencing and collaboration technology, the BMI has asked for meeting rooms to let groups of users from within the ministry or between different ministerial agencies collaborate with each other. So they were provided with a number of electronic meeting rooms (so called team rooms) equipped with large projection units coupled to the same kind of
177
Figure 1 : Variations of technical solution
workstations as already used for the desktop users. Different configurations achieved in the initial solution are shown in figure 1. As a basis for the audio/visual communication there were two different kind of networks: a governmental owned ISDN network (BBN) as well as a governmental owned IP-network (IP-Backbone). Because of the fact that currently every ministry
178
has access to the BBN but not to the IP-Backbone, ISDN has been chosen as the basis for synchronous communication. A further and by far more important reason was the fact that ISDN provides guaranteed bandwidth and delay which are necessary to provide the audio and video quality the users have asked for. For the desktop solution, a bandwidth of 128 kbps was deemed appropriate, while the large projection unit in the team rooms, together with the requirement of hosting discussions between a number of users at each site required higher bandwidths (384 kbps), in order to provide better video and audio quality for all users in the room. Hardware Codecs with integrated ISDN connectivity are used in order to achieve high performance. Their compliance with the international ITU-T standards H.320 2 and T.120 3 ensures interoperability with ISDN-based conferencing systems from other vendors. In order to provide the required multi-point communication, an MCU (Multi-point Control Unit) has been installed which is able to handle a large number of simultaneous conferences, each between several sites. The MCU manages multipoint conferences by distributing the A/V data between the connected sites. It integrates multiple A/V data streams into a single stream for each site which could, for instance, display multiple sites on one screen, e.g. in the form of a quad-split display showing four remote sites at the same time. Assessment of Initial Solution
After the installation and setup of the off-the-shelf components some of the previous mentioned needs of the users could already be met, but there were still several remaining requirements which necessitated the development of the system presented in this paper. With the provided video conferencing system it became possible for all users (even for those who had no connection to the IP-Backbone) to exchange documents in electronic form faster than before. But what the users were missing was a document repository into which they could file their documents for retrieval within as well as between collaborative sessions. The repository should allow structuring the document base by work groups and restrict the access to those work groups or task forces. Many of the necessary steps to establish collaboration between two or more users are purely administrative or consist of application access and application control, tasks which the integrated collaborative application could easily perform for the users. Additionally, many of the necessary steps are performed by different applications, each providing its own distinct user interface, forcing the users to be familiar with a large number of applications just in order to perform the task at hand. Instead, the envisioned system should be provided with enough information beforehand, so that arranging a collaboration session between a number of users would be ideally reduced to merely selecting the document(s) to be cooperatively processed and the user(s) with whom to collaborate. All administrative tasks, the scheduling and reservations as well as the timely invocation of the required tools were to be
179
controlled by the system, thus giving the users the freedom to concentrate on the main task at hand: the joint processing and creation of documents. A major requirement of the users was the provision of a single integrating user interface to control all the different functional modules of the application suite. The off-the-shelf components were neither integrated nor did they provide a homogeneous user interface with a familiar and consistent look and feel to the users. A general problem with today's MCUs is that they do not offer any kind of (standardized) API (Application Programmers Interface), rather they require an operator who carries out conference reservation and control instead of the users. This resulted in the requirement to develop a conference reservation and control system which allows users to create and control audio/video and data conferences themselves. Another problem with today's MCUs is that ad-hoc communication is almost impossible, because of the way in which an MCU conference is set up: When a user decides that he needs a multi-point conference he has to ask an MCU operator to set it up at a certain day and time. This can be very time-consuming (it may take several minutes to hours depending on the availability of the operator). Also, the way an MCU controlled conference is typically set up results in the problem that a multipoint audio/video conference can not be resized dynamically in terms of time and number of participants, i.e. during a running conference, no further users can participate unless that their later participation was already anticipated when making the necessary reservation. An analysis of the remaining unmet requirements led to the specification and development of a Cooperative Document Repository (CDR), into which the initial components were to be integrated and which provided a consistent and easy to use interface to the available functionalities.
Architecture of the Cooperative Document Repository The CDR is a distributed collaborative system, providing the users with a cooperatively accessible document storage as well as immediate information about other users' actions. The architecture is centered around a database server which provides persistence of documents and data objects as well as propagation of realtime update notifications. This central database is provided using a commercially available RDBMS accessed via the ODBC 4 database access layer. The use of a standard RDBMS and ODBC as an access interface was chosen in order to meet the adaptability requirement, as it does not limit the system to a specific DBMS and thus ensures applicability in a wide range of user communities. The central database stores all information required by the system: the document management metadata, the infrastructure and group modeling data as well as the actual document contents. For the Cooperative Document Repository a central data model was developed, based on previous work on a file and document model for the public administration 5 performed within the project. The original object model was revised to address the specific requirements introduced by the collaborative settings to be supported by the project's application goals, especially the stronger
180
need for group and access right modeling. The CDR object model contains all relevant information about not only the contents of the document repository, but also about the system environment, the users accessing the repository and the conferencing equipment used at the different sites. The client applications of the distributed system were developed in Java, using the JDBC 6 interface as a means to access the database server. Java was chosen as an implementation language in order to make use of the existing and future infrastructure within the ministries. The use of the Java programming language and the JDBC database access mechanisms ensures operability of the solution over the governmental IP backbone and also enables, at a later stage, the integration of CDR modules as general services into the currently evolving governmental Intranet. The use of the standard database access mechanism JDBC provides a large degree of flexibility in the selection of the database system used within the system as well as the actual driver library used for database access. The database access module is configurable in order to allow easy exchange of database drivers, which also aids the evaluation of the performance characteristics of the different access libraries. As an example for this flexibility, two different JDBC packages were used interchangeably and evaluated in the course of the development of the CDR. The truly cooperative use of the CDR in the context of a synchronous audio/video conference necessitates the synchronous coupling between the users' client applications. This is performed through instantiation of the CDR object model in the clients in the form of a distributed shared memory. The clients' data sets are automatically kept consistent through the use of the underlying database's transaction system and a combination of object invalidation and active notifications. Client side operations are gathered into transactions, communicated to the database server, whose responsibility it is to ensure transactional consistency. Objects modified within a transaction are identified and an active notification component, based on trigger mechanisms available in the database server, automatically distributes the changes to all connected clients by invalidating the locally cached object replicas and notifying the client applications of these changes. The automatic notification leads to a redisplay of those changed data objects which are currently displayed on the user's screen, thereby providing interactive synchronous collaboration. The client applications contain interfaces to external components such as A/V conferencing, application sharing and integration of a scanner for documents in paper form. Many of the off-the-shelf components used in the project, such as the application sharing system or the videoconferencing system, merely provide C++ programming interfaces which had to be made available to Java applications in a consistent manner. In order to ease flexibility and exchange of components at a later stage, generic Java interface modules were specified for the individual functional modules which abstracted from the actual programming API and presented the module's functionality in an abstract and reusable manner. These interface modules were then mapped onto the interfaces available in the application modules currently used. This abstraction layer fulfills the adaptability requirement, since it effectively
181
hides the set of third-party cooperation and communication tools actually used and facilitates easy exchange of system components. In order to fulfill the requirement for supporting existing document processing applications and enable the cooperative work on documents created and maintained with non-cooperative applications such as the familiar office suites, the CDR provides an integration with external applications (e.g. a text processing system) as well as with third-party application sharing packages. This is achieved through an abstract application integration layer which can be tailored to support a wide range of applications for different document types as well as a number of application sharing packages, accessed through their API functionality. The integration with the application sharing package allows automatic launching of the appropriate application along with the application sharing module within a conference. Due to its knowledge of the current interaction situation and the set of connected users, the CDR can launch the relevant tools and applications in a mode which is required for the current communication setting. To support the requirement of integrating paper-based documents cooperative work processes, scanners and printers are integrated into Documents can be scanned and directly imported into the CDR by way of compatible scanning device, thus making them directly available to collaboration partners.
into the the CDR. a TWAIN the other
For multipoint conferences a reservation has to be made on the MCU. The functionality for making such an MCU reservation is integrated into the CDR. The control and scheduling of the Multipoint Conferencing Unit is performed via the MCU's reservation database, since the MCU itself does not offer a directly usable API. For this task, a specialized daemon was developed which mediates between the MCU's reservation database (resident on a dedicated server machine) and the CDR database. Any relevant changes performed in the CDR database (e.g. the scheduling of a new conference) is immediately fetched and transferred to the MCU reservation database. This daemon interfaces to the CDR server just like the other clients, by way of the distributed shared memory which is automatically kept in a consistent state when changes occur, thereby informing the daemon that changed data needs to be propagated to the MCU reservation database. The architecture diagram in figure 2 shows the "building blocks" of the CDR application. The shaded boxes denote the modules developed in Java, boxes with emphasized borders denote individual applications within the CDR.
182
Figure 2: Architecture Diagram of CDR
183
Usage and Functionality of the Cooperative Document Repository The repository front-end is a group-aware application: it not only provides the users with a view of the repository's contents, but also displays information about which users are currently cooperatively viewing or editing which documents. This groupawareness is provided in the form of name tags added to the repository content displays, indicating current cooperation activities on the documents (see figure 3). In order to be immediately applicable by the users, the CDR provides a document management paradigm similar to the file management structures already familiar to the users from the Windows environment. The CDR enables the users to structure the cooperatively used document space into individual repositories, containing a hierarchy of folders, which in turn contain the documents. The access rights and group modeling mechanisms present in the CDR allow the users to control and manage their group cooperation and restrict other users' access to the documents in the CDR. Every user of the Cooperative Document Repository can create groups and repositories to match his or her current requirements. For a task which is to be tackled cooperatively, a new group within the system can be created, including those users which require access to the common documents. Along with this, a new repository can then be set up, using group- and user-based access permissions to control which users and which user groups may view or edit the documents contained in the repository. The newly created repository (or, rather, an icon depicting it) automatically appears on the CDR desktops of all users who are allowed to access the repository and who are currently logged into the system, allowing them to begin their work immediately. The solution is kept scalable to a large number of users by only presenting those repositories and repository contents at the GUI to which the current user has access. The group members can now cooperatively begin to structure and fill the repository, by creating folders as required and importing documents relevant for the task at hand. In this way, the group members are provided with a common, cooperative document storage system which they can use to store and exchange documents related to the task at hand. The fact that all users have concurrent access to the CDR simplifies exchange of documents.
184
Figure 3 : Desktop view of CDR
It is important to note that the CDR was not intended to be a replacement for classical document management and storage systems. All tasks involved in the strict document management methods required in the ministerial context, such as registering all documents entering and leaving the ministry, managing consistent document and file numbers, maintenance of a filing plan, etc. are outside the scope of the CDR. All official document management functionalities will continue to be provided by dedicated document management systems (DMSs), which are approved for use in the ministerial environment and which potentially differ from ministry to ministry. The structuring means and functionalities provided by the CDR are driven by the requirements of the task-oriented and flexible cooperation activities within the user community. The CDR does not enforce a strict organizational model of the individual ministries, departments, etc. The cooperation structures encountered in the application domain do not rely on these organizational models but instead involve work groups spanning organizational bodies and set up for short- to medium-term activities. Also, the CDR is used as the unifying basis for more informal cooperations such as unplanned ad-hoc activities. These informal collaboration tasks would more often be hindered than simplified by adherence to too strict an organizational model.
185
Document-Based Collaboration To give an impression of the functionality of the CDR and to provide an illustration of how the integrated system facilitates the cooperation processes, a short scenario is presented in the following. Suppose a user is member of a working group which is to come up with a document on the security of firewall technology for protecting the Governmental IP backbone. Assume that this user has been given the task to prepare an initial draft for this document. The cooperative scenario could, e.g., look like this: 1. In the CDR the user opens the repository that has been set up for the working group in question. With a double-click on the document the associated external application (here: a text processor) is started from within the CDR and the user can carry out his task of preparing a draft. 2. In the course of his work he might reach a point where a direct discussion with other members of the working group becomes necessary. From the list of group members the user selects his collaboration partner(s), indicates to the system that the document he is currently working on is to be subject of the collaboration and then simply invokes a ,,collaborate" command from a pop-up menu within the CDR. The CDR then automatically performs all steps which are necessary to set up this document-based ad-hoc conference: a) An audio/video conference is set up between all collaboration partners. To enable a multipoint conference a corresponding conference is registered on the MCU. The MCU can then call all participants and establish the multipoint A/V conference. All data necessary for this technically complex step is available in the system (the telephone numbers, the types of the involved end points, etc.). After they accept the call, the video window is finally opened on each of the users' desktops. b) The application sharing package that is needed to cooperatively edit the document is started for every participant in the conference. Here again, all technical information (e.g. connection numbers) is available in the system and is used to set up the sharing session without any manual interaction by the users. The text processor that is to be used cooperatively appears on the desktops of the conference participants. 3. Now that the working group has come together, the draft preparation can continue. The contents can be discussed over the audio/video connection and changes or extensions can be entered by any of the participants. 4. Users can easily input paper documents to the system by using the integrated scanner solution. The scanned document is directly imported into the group's document repository. Documents can of course also be printed if wanted.
186
5. The participants can leave the conference at any time individually. All related connections are closed automatically and finally the whole conference is shut down. This example illustrates that working on documents in the repository as well as seamlessly switching from single user work to multilateral cooperative tasks requires only a minimum of user actions while the system can perform all complex ,,supporting actions" automatically. In the following, some features of the CDR are presented in more detail.
Initiation of Cooperative Sessions In order to cooperatively process a document from the CDR together with other users of the system, ideally all the user needs to indicate to the CDR desktop is which document he wishes to process together with which other users. Since the system is provided with all the necessary information about the users' infrastructure, such as the phone numbers of the ISDN videoconferencing system available to the user, the supported bandwidths, etc., it is able to schedule and establish a collaboration session without further user input - apart from the obvious interaction steps of confirming the communication partners' readiness to enter the collaboration session now or to postpone it to a later point in time. Since the documents which form the basis for the collaborative activity are already present in the CDR, distribution of these documents also becomes an easy task.
Conference Scheduling A reservation system is available which generates all technical entries required by the MCU database and manages reservations of the required team rooms. A small part of the data needs to be input by the user (names of the participants, the date, time and duration of the conference etc.), while a major part of the information is stored in and retrieved from the CDR database (e.g. telephone numbers for the A/V connection, type of A/V hardware). Thus, an operatorless operation of the MCU is possible. Should no MCU resources be available for the chosen time slot, the user will be immediately informed that the conference is not possible at the chosen time. In addition to the MCU resources, any team rooms that are involved are checked for availability and are then reserved for the new conference. The newly created conference automatically appears in the conference list on the desktops of all conference participants, thus informing them about the new conference. When the time of the conference arrives, the MCU either automatically dials out to all participants or the CDR notifies the user about a conference being started. As soon as the user has confirmed his readiness the appropriate connections are established (again without requiring the users to remember and dial any numbers etc.). The initiation of ad-hoc conferences is achieved in the same way, simply by scheduling the start of the conference with the current date and time.
187
Integration of External Applications Opening a document within a conference will indicate the user's desire to present this document to the entire group of connected users or to edit it cooperatively with them. The CDR supports this activity by automatically launching not only the required application but also the application sharing component - if necessary, on all connected machines - and bringing the document's application into the application sharing session ready for collaboration. This support relieves the users of a large part of the burden involved in setting up the conference connections, the required applications and the application sharing system. Should a document be editable or presentable by a truly collaborative software such as the DOLPHIN 7 system, which does not require the additional application sharing support, this integration is also available in the CDR. Again, since the CDR has access to all the necessary information about the collaboration situation, the external collaborative application can be invoked on all sites, receiving the required parameters for connection and setup from the CDR.
Summary Using the integrated collaboration environment provided by the CDR in conjunction with the integrated communication and collaboration tools as presented above, the users can initiate and conduct their collaboration tasks in a manner which is appropriate to the problems they actually need to tackle. Indicating the collaboration partners and the document(s) on which to collaborate is sufficient for the system to set up the required communication channels, make the required reservations and start the necessary tools at the right point in time. The users are relieved of the effort of having to control (and learn) a number of separate applications and can concentrate on the actual work processes.
Related W o r k A number of research and development projects have taken on the task of supporting document-based collaboration between multiple distributed sites. This section compares representatives of these systems to the requirements listed in this paper and addressed by the CDR. The BSCW shared workspace system 8 is a Web-based groupware application which follows a collaboration paradigm similar to the CDR: users can create and structure repositories into which they can import documents which are then made available to all users who currently have access to the repository. The documents can be retrieved by the users, processed and at a later stage put back into the BSCW system for the other users. As a purely Web-based application it differs from the CDR in a number of points which were of importance in our application context (as shown in the requirements section). Firstly, BSCW is an asynchronous system which does not provide direct feedback and information about other user's actions (synchronous group awareness; R2.1). In order to observe changes within the repository, the Web page has to be manually reloaded by the user. Also, BSCW provides no integration of synchronous multi-party A/V communication (R2.2 and
188
R2.3). The CoopWWW 9 project aims at extending the BSCW system through the integration of A/V conferencing between the system's users. It currently does this by integrating a software A/V communications module, CUSeeMe 10. The resulting system does not offer the high-quality A/V conferencing supplied by the CDR with the integrated ISDN conferencing system. Since CUSeeMe is a software solution and employs the Internet for data transport, the available bandwidths and thus the achievable frame rates and picture quality are much lower than the ones achieved by integrating ISDN-based multipoint conferencing as we have done in the CDR. An evaluation of user requirements has shown, though, that audio and video quality are of utmost importance for the acceptance and wide-spread use of the resulting technical solution (R2.2). Lotus Realtime Notes 11 is a combination of Intel ProShare and Lotus Notes. Notes, as a replicated distributed document database can be used to store documents which are accessible to a group of users. The use of Intel ProShare allows real-time communication and collaboration on these documents by use of the application sharing component. What this combination does not achieve, though, is the seamless integration of multipoint communication and multi-party distributed settings (R2.3). Also, this combination does not provide an integrated user interface with a familiar look-and-feel (Requirements R3.1, R3.2 and R3.3). The CDR goes one step further towards a complete integration by integrating scheduling, control and reservation of multiparty conferences. Furthermore, Realtime Notes does not provide synchronous group awareness to the system's users. Neither Realtime Notes nor BSCW/CoopWWW are adaptable to different standardsbased collaboration and communication tools in the way in which the CDR is (R5.1). The TeamRooms system 12 also aims at supporting collaboration of distributed teams by providing persistent repositories for the collaboration documents. TeamRooms, though, does not integrate synchronous multimedia conferencing (R2.1, R2.2, R2.3) and uses specialized "applets" for cooperative document creation, thus has no provision for legacy applications and seamless integration into the collaboration processes (R1.1, R1.2).
Conclusions and Future Work This paper has presented the design and implementation of the CDR application, a system designed to support cooperative document management and document processing in a ministerial or governmental setting. It facilitates group cooperation and communication by integrating existing off-the-shelf components into a single comprehensive user interface which allows easy control of the various functionalities available in the different components, which were previously not integrated and therefore complex to employ in a productive manner. The adherence to the metaphor of document-based collaboration as a means of interacting with the system and performing the cooperative tasks has been implemented after an analysis of the collaborative tasks within the context of distributed governmental agencies.
189
Even though the design and development of the CDR was strongly driven by its current application context of distributed governmental agencies, we believe the system is flexible and powerful enough to be usable in a wide range of application domains even beyond the public administration sector. Its approach of supporting the cooperation and communication tasks involved in joint document creation and document-based meeting support as well as its foundation on commercially available system components make it usable in a large number of distributed settings, e.g. in virtual organizations or distributed teams. The CDR has been installed in the user community and we are currently gathering feedback from the users which will directly influence the further development of the collaboration solution. Possible extensions of the system include enhanced document management functionalities, e.g. the support of different document versions within the repository as well as advanced document indexing and retrieval features. An extension of the architecture to include several distributed database servers might become necessary in order to better support the distributed setting and address scalability issues. Other aspects that will be examined in the ministerial application context is the use of secure audio/video conferencing, e.g. by using video encryption, as well as an advanced authorization concept for the team room setting to enable the use of the CDR application suite also in security-sensitive environments.
Acknowledgements The authors would like to thank the following people for the effort they put into the development of the software described in this paper: Marc Volz, Bernd Bussek, Oguzhan Karaduman, Fritz Lorenz, Sascha Ltidecke, Michael Pauly. We are also grateful to J6rg Haake for valuable comments about preliminary versions of this paper.
References 1
Engel, A., Kaack, H., Kaiser, S.: Teamarbeitsraume zur Untersttitzung verhandlungsorientierter Telekooperation; in: Rechnergestiitzte Kooperation in Verwaltungen und groflen Unternehmen, Tagungsband zum Workshop der GI-Fachgruppe FG 551 und der GI-Fachbereiche FB6 und FB8 im Rahmen der Jahrestagung der Gesellschaft ftir Informatik; Aachen, 1997
2
ITU-T Recommendation H.320 (03/96) - Narrow-band visual telephone systems and terminal equipment; available electronically from http://www.itu.ch/itudoc/itu-t/rec/h/h320_23397.html; 1996
3
ITU-T Recommendation T.120 (07/96) - Data protocols for multimedia conferencing; available electronically from http://www.itu.int/itudoc/itut/rec/t/t120_35511.html; 1996
4
Microsoft ODBC 3.0 Software Development Kit and Programmer's Reference; Microsoft Press, 1997
190
5
Knopik, T.; Tietze, D.; Volz, M,; Paul, B.; Heite, R.; Speichermann, H.; Wittinger, C.: Towards a Collaborative Document Archive for Distributed Governmental Agencies; in: Lehner, Dustdar (eds): Telekooperation in Unternehmen; Dt. Univ.-Verlag, Wiesbaden, Gabler, 1997; pp. ~
6
Graham Hamilton, R. G. G. Cattell, Maydene Fisher: JDBC Database Access With Java: A Tutorial and Annotated Reference (Java Series); AddisonWesley, 1997
7
Streitz, N.A., GeiBler, J., Haake, J.M., and Hol, J. DOLPHIN: Integrated Meeting Support across LiveBoards, Local and Remote Desktop Environments. In Proceedings of the 1994 ACM Conference on Computer Supported Cooperative Work (CSCW '94), Chapel Hill, N.C., October 22-26, 1994, pp. 345-358.
8
Appelt, W.; Busbach, U.: The BSCW System: A WWW-based Application to Support Cooperation of Distributed Groups. In: IEEE (ed.): Proco of the Fifth Workshops on Enabling Technologies: Infrastructure for Collaborative Enterprises, June 19-21, 1996, Stanford, CA. Los Alamitos, CA: IEEE Computer Society Press, 1996, p. 304-309.
9
project description available at http://orgwis.gmd.de/projects/COOPWWW/
10
Information about CUSeeMe is available electronically http://bio444.beaumont.plattsburgh, edu/CUSeeMe, html.
11
Information available electronically from Lotus Realtime Notes Web site at http://www.lotus, com/home.nsf/welcome/realtime
12
from
Roseman, M., Greenberg S.: TeamRooms: Network Places for Collaboration; in: Proceedings of CSCW'96 (ACM Conference on Computer-Supported Cooperative Work); Cambridge, MA, USA; 1996
Specifying the Reuse Context of Scenario Method Chunksl Colette RoUand*, V6ronique Plihon§ Jolita Ralyt6* *Universit6 Parisl-Sorbonne CRI 17, rue de la Sorbonne 75231 Paris Cedex 05, France Tel : + 33 (0)1 40 77 46 34 Fax : + 33 (0)1 40 77 19 54 {rolland, ralyte}@univ-paris 1.fr
+Universit6 de Toulon et du Var GECT BP 132 83957 La Garde Cedex, France Tel : + 33 (0)4 94 14 25 41 Fax : + 33 (0)4 94 14 21 65 [email protected]
: There has been considerable recent interest in scenarios for accompanying many of the various activities occurring in the development life cycle of computer based systems. Besides the integration of scenarios in methods such as Objectory and software tools such as Rationale Rose has proven useful and successful. Consequently, there is a demand for adapting existing methods to support specific design activities using scenario based approaches. The view developed in this paper is that scenario based approaches should be looked upon as reusable components. Our concern is therefore twofold: first, to represent scenario based approaches in a modular way which eases their reusability and second, to specify the design context in which these approaches can be reused in order to facilitate their integration in existing methods. The paper concentrates on these two aspects, presents an implementation of our proposal using SGML to store available scenario based approaches in a multimedia hypertext document and illustrates the retrieval of components meeting the requirements of the user by the means of SgmlQL queries. Abstract
1. Introduction Scenario based approaches have proven useful in a large number of situations occurring in the system development life cycle. In the HCI community, scenarios have been proposed as detailed descriptions of a usage context so design decisions can be reasoned about Carol195 or as small examples of an existing product which are used to anchor discussion about different design theories Young87. In Software Engineering, use case approaches have been developed to derive the object oriented specification of a system from narrative descriptions of interactions with its users. In the Information Systems community scenarios have evolved to the concept of a rich picture which gives the social setting of a required system so arguments can be developed about the impact of introducing technology, and the matching between user requirements and task support provided by the system Kyng95. Finally in 1This work is partly funded by the Basic Research Action CREWS (ESPRIT N~ CREWS stands for Cooperative Requirements Engineering With Scenarios.
192
Requirements Engineering, scenario scripts based approaches have been proposed to support the checking of dependencies between a requirements specification and the user/system environment in which it will have to function Potts94. These examples demonstrate that a scenario based approach aims primarily at supporting some specific design activity. By essence these approaches are not standalone products but instead, they have vocation to be integrated in existing methods to support some specific steps of the design process with the advantage of increasing usability. As a specific scenario based approach provides support to a specific design activity, it might be possible to integrate it in various different methods dealing each with this particular design activity. Our view is that scenario based approaches should be looked upon as reusable components. This reuse perspective has been already illustrated, for example by the use case approach originally developed by Jacobson Jacobson95a, Jacobson95b, and then, integrated in a number of existing methods including the Fusion method Coleman94, OMT Rumbaugh91, Rumbaugh94 and UML Booch97. However reuse has been performed in an 'ad hoe' manner while there is a demand Jarke97 for a more systematic way of understanding when, why and how, which kind of scenario has to be used. Thus, if we want to support the reuse of the large corpus of available scenario based approaches we shall solve the problem of characterising the context in which they can be reused. In this paper we are concerned by these two issues : (a) to represent scenario based approaches as reusable components and (b) to specify the context of use of available scenario based approaches in order to facilitate their reuse in different methods to support the design activities they are dedicated to. Our proposal is based on an analogy with object oriented reuse and comprises two aspects: 1. To define a scenario based approach as a collection of methods fragments that we call scenario method chunks (scenario chunks for short) and to make them available in a scenario method base. 2. To characterise the context of use of scenario chunks in chunks descriptors and to store them in the scenario base together with the chunks themselves. This shall ease the retrieval of scenario chunks meeting the requirements of the method base user.
Our view is therefore to organise the scenario method base at two levels, the method knowledge level and the method meta- knowledge level and to tightly couple scenario chunks with the meta-knowledge describing their context of use. The method level tells us how to apply a specific scenario chunk whereas the meta-level provides knowledge about the conditions under which the scenario chunk is applicable. Our notion of chunk descriptor is close to the view of De Antormellis91 and very similar to the one of faceted classification schema Pietro-Diaz87 developed in the context of software reuse. We have implemented the proposed approach using SGML (Standard Generalised Markup Language). The two knowledge levels of the method base are parts of the
193
same SGML document in order to facilitate their joint manipulation. Our motivation for using SGML has been on the one hand, its ability to represent hyper-text documents and on the other hand, the availability of sgmlQL which is an SQL like language tailored to query SGML documents. This provides the required facilities to query the scenario method base and retrieve the scenario chunks which match specific reuse conditions. In the rest of the paper, we develop in detail the scenario method knowledge and the scenario meta-method knowledge as well as their implementation. Section 2 deals with the former, presents the notion of scenario chunk, illustrates the different levels of granularity of chunks and exemplifies them by describing several existing scenario based approaches. Section 3 deals with the meta-knowledge representation, defines and exemplifies the notion of chunk descriptor. Section 4 covers the implementation of the scenario method base in SGML. In section 5 we illustrate through examples of queries in sgmlQL how scenario chunks can be retrieved from the method base. Finally we draw some conclusions in section 6. 2. S c e n a r i o M e t h o d K n o w l e d g e L e v e l We adopted a modular approach to represent the scenario method knowledge, in the method base, in the form of scenario method chunks, scenario chunks for short. A scenario chunk may represent an entire approach such as the Jacobson's use case approach or part of it, for example the chunk to define abstract use cases. This eases the reusability of chunks and their integration in methods. A chunk tightly couples a product part and a process part. In the product part, the product to be delivered by a scenario chunk is captured whereas in the process part, the guidelines allowing to produce the product are given. As we are interested in scenario chunks, at least one of the product parts involved in a chunk must be of the scenario type. The guidelines to defme the use case model proposed in the OOSE methodology Jacobson92, to capture and use scenario scripts Potts94, to construct interaction diagrams Rumbaugh96, or abstract usage views Regnel195 are examples of such scenario chunks. 2.1 The Notion of Scenario Chunk
Our definition of a scenario chunk is based on the process view of the NATURE process modelling formalism Rolland94, Plihon95 and consistent with the notion of 'step' in Thorn693 . According to this view a process can be seen (Fig. 1) as a black box which transforms an initial situation into a result which is the target of the intention of the process. The situation represents the part of the product undergoing the process and the intention reflects the goal to be achieved in this situation. The target of the intention is the result produced by the process execution. As the target is embedded in the intention, this leads to the characterisation of a process by a couple <situation, intention> which is called context (see the example in Fig. 1).
194
Fig. 1. The behavioural view of a scenario chunk
Following this view, a scenario chunk has two parts (Fig. 2 ) 2 ; its interface which is the couple <situation, intention> and a body. We chose these designations by analogy with object descriptions in object oriented approaches. The interface is the visible part of the chunk. It tells us in which situation and for which intention the chunk is applicable. The body explains how to proceed to fulfil the intention in that particular situation. The body provides guidelines to guide the process and relates the process to the product parts involved. For example, the interface of the scenario chunk representing the Jacobson's use case approach is the context <(Problem Statement), Define Use Case Model > where the situation is represented by a document called problem statement and the intention, to define use case model, is based on this problem statement. The target of the intention, use case model defines the result to be produced by the application of the chunk. The body provides the guidelines to achieve the intention of defining a use case model out of problem statements (see Fig. 4 that we will explain later on).
Fig. 2. The scenario chunk structure
Additionally, as shown in Fig. 2, each scenario chunk in the method base has a unique name. It may be represented graphically and/or described informally in natural language. It is also possible to provide some examples of anterior application of this scenario chunk. A scenario chunk is classified either into formal or informal. In the former case the guidelines are formally deemed whereas they are informal textual recommendations in the latter.
2 The structure description is based on E/R like notation.
195
As illustrated in the example above, the intention contains the reference to the target. In fact the structure of an intention (Fig. 3) is more complex than a simple verb. The proposed formalisation of the notion of intention permits a t-me grain characterisation of the chunk which was felt necessary to support efficient retrieval of scenario chunks.
Fig. 3. The intention structure
The intention is decomposed [Prat97] into a verb, a target (a product part) the verb is acting on and a manner. Depending on the role played by the product part for the verb, we make the distinction between objects and results. An Object exists in the situation of the corresponding context whereas a Result is produced by the achievement of the intention. "Refine use case", is an example of intention in which the target "use case" is an object because it already exists in the situation whereas "Identify actor" is an example where the target "actor" is a result. It is developed during the execution of this intention. The precise notation of these intentions is as follows: <(Use Case), Refine (Use Case)Obj>, <(Problem Statement), Identify (Actor)Res>. In addition to the verb and the target, the intention may have a manner which specifies in which way the intention is satisfied. A manner is a strategy or an approach used to fulfil the intention. ~One-shot refmemenb) or ~stepwise strategy)) are examples of manners. The proposed definition of a scenario chunk is applicable to any method chunk. The distinction between a scenario chunk from any other method chunk is due to the nature of the product parts. In the latter the product can be of any type whereas in the former either the situation or the target of the intention must refer to a product part of the scenario type. For example, the two chunks with the following interfaces <(Problem Statement), Define (Use Case Model)Res> and <(Use Case Model), Interpret <(Use Case Model)Obj> are scenario chunks. Both manipulate scenarios which are called use cases, the former having the use case model as the target of the intention to Defme Use Case Model whereas the use case model is the object of the Interpret intention. The application of the NATURE contextual approach to the representation of method knowledge has the important effect of making chunks modular. A chunk prescribes the way to proceed in a situation to fulfil an intention. A scenario chunk can be qualified as cohesive because it tells us the situation in which it is relevant and the
196
intention that can be fulfilled in this situation. A chunk is loosely coupled to other chunks because it can be used in the appropriate situation (created as the result of another module) to satisfy the intention. Thus, the linear arrangement of method modules is replaced by a more dynamic one. Finally, the hooks to combine a scenario chunk with another chunk (whichever is its type) are parts of its interface: the situation and the target. Two chunks can be assembled if the situation of one of them is compatible with the target of the other. 2.2 The Body of a Scenario Chunk The interface of a scenario chunk characterises the conditions of its applicability whereas its body details how to apply it. The interface plays a key role for retrieving a scenario chunk out of the method base while the body is used when applying the method in which the chunk has be integrated. Our approach relies upon the interface structure presented above but does not imply a particular way of describing the chunk body. In the sequel, we illustrate partially the solution we chose for the implemented method base.
Fig. 4. Excerpt of a scenario chunk [Jacobson921
We follow the NATURE approach and define the chunk body as a hierarchy of contexts called a tree. As illustrated in Fig. 4 contexts relate one to the other through three types of links : refinement links which permit the refinement o f a large-grained context into finer ones, composition links which allow to decompose a context into component contexts and action links which relate the contexts to the actions which directly perform transformations on the product. Each type of link has a
197
corresponding type of context, namely executable, choice and plan contexts. A detailed description of contexts can be found in Rolland96. Let us briefly illustrate them through the example of tree presented in Fig. 4. A choice context offers choices supported by arguments. The context <({Use Case}), Refine Use Case> in Fig. 4 introduces three alternatives to a Use Case refinement : (1) to generalise a set of use cases into an abstract use case <({Use Case}), Generalise {Use Case} into Abstract Use Case>, (2) to specialise an abstract use case into concrete use cases <(Abstract Use Case), Specialise Abstract Use Case into Concrete Use Case> or (3) to extend a use case with a use case extension <(Use Case), Complete Use Case with Use Case Extension>. A plan context corresponds to an intention which requires further decomposition. The plan context <(Problem Statement), Define Use Case Model> Fig. 4. is composed of three contexts, namely <(Problem Statement), Identify Actor>, <(Problem Statement, Actor), Define Use Case> and <({Use Case}), Refine Use Case>. This means that, while constructing a use case model, the requirements engineer has first to identify the actors, then to construct use cases, and finally, to refine the use cases. The component contexts of a plan context can be organised within graphs in a sequential, iterative or parallel manner (see the top bubble in Fig. 4). An executable context corresponds to an intention which is directly applicable through actions which transform the product under development. Performing the action changes the product and may thus generate new situations. In Fig. 4 the context <(Problem Statement), Identify a Primary Actor> is an executable context. The intention to "Identify a Primary Actor" is immediately applicable through the performance of an action which creates a new "primary actor" in the use case model under development. Notice that when the chunk is informal, its body is reduced to a textual explanation on how to perform the action (see Fig. 5). Contexts can therefore be related through refinement and composition links to form a tree. A tree is considered complete when all its leaves are executable contexts. The scenario chunk in Fig. 4 is a tree in which the root context <(Problem Statement), Define Use Case Model> is decomposed and refined throughout the hierarchy of contexts to the set of executable contexts. Thus, the global intention to defme use case model is transformed into a set of actions which transform the product to develop the use case model. For the sake of clarity, the tree is only partially represented in Fig. 4. Finally, our concrete experience in constructing the SGML scenario knowledge base has raised difficulties due to the lack of formal descriptions of either the process or the product models of the scenario approaches. Moreover, approaches rarely include the process dimension. To cope with this situation, we classify scenario chunks into formal and informal (see Fig. 2) and propose to associate to informal chunks textual explanations on how to fulfil the intention of this type of scenario. For example, in Fig. 5 we present a scenario chunk defined according to Kyng's approach Kyng95. This chunk proposes to describe work situations in which the users identify some problems and bottlenecks. The author does not detail how to work with these scenarios, he only explains what type of information these scenarios have to contain.
198
Fig. 5. Scenario chunk from Kyng's approach 2.3 Scenario C h u n k Granularity
Fig. 6 sums up the three different levels of granularity of scenario chunks: contexts, hierarchies of contexts called trees which may be parts of a scenario approach. Each of these chunks can be considered either independently or as part of an overall scenario approach. A scenario based approach is viewed itself as a chunk (is-a link in Fig. 6). Indeed, both the approach itself and its component chunks are reusable. Typically, a scenario approach contains guidelines for the creation, the transformation and the refinement of scenarios into more conceptual products. For example, in the OOSE methodology [Jacobson92], we identified two scenario chunks, one to construct the use case model and a second one to construct the analysis model out of the use case model. The composition of these two chunks corresponds to a scenario approach which is also proposed in the method base as another chunk.
Fig. 6. Structure of the scenario knowledge in the scenario method base
199
3. Scenario Method Meta-Knowledge Level The scenario method knowledge is about descriptions of available scenario method chunks. The scenario method recta-knowledge we are dealing with in this section aims at specifying the context in which method knowledge can be (re)used. Assuming that the scenario base has been constructed, the question addressed now is ~ how to ease the retrieval of scenario chunks meeting the requirements of a method engineer who wants to extend an existing method with scenario features?)). This raises the need for understanding when, why and how a specific scenario chunk can be reused i.e. to specify the context of its use. Our literature survey Rolland97 as well as the industrial visits performed within the companies of the CREWS steering committee Jarke97 have shown that this knowledge is not available. Both have also demonstrated that there is an explicit need for making this knowledge available. Our view is that the knowledge about the context of use of scenario chunks shall be formalised and stored in the scenario method base with the scenario chunks themselves. We call this knowledge method recta-knowledge as it provides information characterising the use of scenario method knowledge. The scenario method base is therefore organised at two levels, the method meta-knowledge level and the method knowledge level. In the process of reusing scenario chunks, these two levels serve in separate steps. The method meta-knowledge supports the retrieval step whereas the knowledge is the material effectively reused and integrated in the existing method. In this section we are concerned with the meta-knowledge representation. We shall illustrate the use of this meta-knowledge in section 4 through sgmlQL queries acting on the implemented method base. We use the notion of descriptor De Antonnellis91 as a means to describe scenario chunks. A descriptor plays for a scenario chunk the same role as a meta-class does for a class. Our concept of descriptor is similar to the one of faceted classification schema Pietro-Diaz87 developed in the context of software reuse. We extend the contextual view used to describe the chunk interface to structure the meta-knowledge in the descriptor. Indeed, we view the retrieval process as being contextual : a user of the method base is faced to reuse situations at which he/she looks with some intention in mind. Therefore, the descriptor seeks to capture in which situation a scenario chunk can be reused to fulfil which intention. If we remember that scenario based approaches primarily aim at supporting specific design activities in different ways, the descriptor situation shall refer to this design activity whereas the intention expresses a design goal related to this activity. As an example, the descriptor of the Jacobson's chunk described in section 2 shall refer to 'analysis' as a design activity supported by the chunk and 'capture user/system interactions' as the intention within this activity which can be supported by the use case approach provided by the chunk. Then, our descriptor is contextual as it captures situational and intentional knowledge deeming the context of reuse a scenario method chunk.
200
Fig. 7 gives an overview of the knowledge captured in the method base for every chunk. The chunk body is actually the fragment of method to deal with a specific type of scenario whereas the chunk interface describes its conditions of applicability, the situation required as input of the chunk, and the intention the chunk helps to fulfil. These two aspects constitute the scenario method knowledge whereas the metaknowledge is captured in the scenario descriptor. The descriptor expresses the reusability conditions of the chunk by characterising the design activity in which it can be reused (the situation part) and the design intention that can be supported by the scenario chunk (the intention part). It describes the broad picture in which the scenario approach captured in the chunk can take place. In the sequel, we develop the chunk descriptor in detail.
Fig. 7. Chunk overview 3.1
The Descriptor Structure
Fig. 8 depicts the descriptor structure. A chunk descriptor has a situation part and an intention part that we consider in turn.
Fig. 8. The chunk descriptor structure The Situation Part of a C h u n k Descriptor
The situation part of a descriptor comprises two aspects (Fig. 8): the application domain and the design activity in which the scenario chunk is relevant. For instance,
201
considering the Jacobson's chunk (Fig. 4) which describes how to proceed for defining a use case model, the domain of the descriptor is Object Oriented Applications and its design activity is Analysis. This means that this chunk can be reused in Object Oriented Application for facilitating the Analysis step. While populating the scenario method base, we have identified a list of application domains in which scenarios are used. The current list of domains in the implemented scenario base is the following: 9 Usability Engineering 9 OO applications 9 Requirements Engineering 9 HCI (Human Computer Interfaces) 9 Workflow applications 9 Critical systems 9 Information systems 9 Socio-technical applications HCI (see Carol195 for a survey) and OO applications Cockburn95, Glinz95, Jacobson92, Lalioti95, Regnel195, Robertson95, Rubin92, Rumbaugh91, Wirfs-Brook95, Leite97 are the two domains where scenarios are nowadays extensively used. Similarly we have identified a list of design activities, (similar to the one proposed in Carol195, Table 1) each of which is supported by at least one scenario chunk. Design Activity Analysis
Scenario Based Approach Carol195,Cockbum95, Glinz95, Jacobson92, Regnel195, Robertson95, Rubin92, Rumbaugh91, Wirfs-Brook95 Envisionment Jacobson 92, 'Nielsen 95,Kyng 95, Karat 95 Requirement Elicitation Holbrook90, Jacobson92, Johnson 95, Kyng95, Potts94 Design Rationale Nielsen95, Kyng95 Validation Holbrook90, Glinz95,Lalioti95, Nielsen95 Software Design Holbrook90,Hsia94 Software Testing Kyng95 Team Work Building Filippidou97 Communication Holbrook90, Jacobson 92, Potts 94, Erickson 95 Documentation / Training Kyng95, Potts94 Table I : Design activities covered by different scenario chunks The Intention Part of a C h u n k Descriptor
The chunk descriptor intention expresses how the scenario approach encapsulated in the chunk participates to the achievement of the design activity. For example, the intention of the descriptor of the Jacobson's chunk presented in Fig. 4 is 'capture user/system interactions' as the chunk provides a scenario based approach supporting the capture of the interactions between the future system and its users. The descriptor intention is an expression of the role that a scenario approach can play in a particular design activity. We found in our survey of both literature and practice a large panel of roles, all being informally expressed and therefore difficult to classify and organise to
202
support the retrieval process in the method base. In the following list we give some examples of our findings: 9 9 9 9 9 9 9
Supporting the analysis of the users workplace and work situations Expressing how a system being designed should look like and behave Facilitating the discovery of user needs Helping evaluating possibilities for usability and functionality Supporting the identification of central problem domain objects Helping to develop cohesion in the team Facilitating the communication on the systems problems between users and designers 9 Helping to test whether the system satisfies or not all the user's requirements 9 Supporting user training 9 Bridging the gap between the system presented as an artefact and the tasks the users want to accomplish using it
Instead of using role names as described in the literature, we use a more formal description of intentions based on Prat97 leading to the intention structure presented in Fig. 8. This structure is compatible with the one used for the chunk interface. It is extended in order to link the intention of the chunk and the intention of its descriptor. The intention in the chunk descriptor is specified by the intention verb, the target of by this intention and the manner to satisfy this intention (Fig. 8). Let us detail these various elements of the intention structure in turn. Similarly to the target in the scenario chunk interface (see Fig. 3), the target of the descriptor is specified into object or result depending on the role played by the target for the verb. These roles have been explained in section 2. Moreover, in the chunk descriptor intention we make the distinction between non-scenario based target and scenario based target (see is a links in Fig. 8) 9 Non-scenario based targets represent product parts other than scenarios. Functional system requirements, non functional system requirements, object model, alternative design option, etc. are examples of non-scenario based targets. 9
Scenario based targets represent product parts of the scenario type. Use case, scenario script, episode, work situation description or use scenario are examples of scenario based targets. In order to ease the retrieval process, there is a need for characterising scenario targets with enough details to differentiate one from the other. Our characterisation is based on the framework defined in the CREWS project Rolland97. Properties such as the scenario formality, the level of abstraction, the nature of interactions, the covered requirements or the scenario life cycle are proved necessary to select more precisely the adequate scenario chunks. There are eleven properties which are surveyed in appendix 1.
The chunk descriptor intention is completed by a manner which is a complex manner by opposition to a simple manner as used in the scenario chunk interface. A complex manner is recursively described as an intention. The intention to Capture user/system interactions by defining a use case model with Jacobson's refinement strategy is an
203
example of descriptor intention using a complex manner. The manner (by defming a use case model with the Jacobson's refinement strategy) is recursively defmed as an intention (defining) with a result ( use case model) and a manner (with the Jacobson's refmement strategy). The intention is denoted as follows: Capture (User/System Interactions)Res (by Defining (Use Case Model)Res (with Jacobson's Refinement Strategy)Man )Man. The descriptor intention always refers to a complex manner. This allows us to link the scenario chunk intention to the descriptor intention. This is modelled in figure 8 by an is a link from the manner to the chunk interface intention. In the example above, the intention to define use case model with Jacobson's refinement strategy is the intention of the Jacobson's chunk presented in Fig. 4. It is embedded in the descriptor intention as the manner of the intention to Capture user/system interactions. The scenario approach captured in a given scenario chunk is formally defmed as the manner to achieve a design intention. Both the design intention and the manner to achieve it are expressed in the descriptor intention. 3.2 Examples In Fig. 9, we present the example of the descriptor corresponding to the Define Use Case Model scenario chunk depicted in Fig. 4.
Fig. 9. Example of the Jacobson's chunk descriptor
This descriptor tells us that the corresponding scenario chunk is useful in the domain of object oriented applications for supporting the analysis activity. The intention in this situation is to capture the interactions between the user and the system. Moreover, the intention is clarified by the manner to fulfil it. This manner defines precisely the way the scenario chunk will help fulfilling the intention of user/system interaction capture: it is by defining a use case model in the Jacobson's style. Because the target
204
of this intention (define use case model) is a scenario based one, the descriptor specifies its discriminant properties, namely the FormDescriptionMedium (textual), the FormDescriptionNotation (informal), the ContentsCoverage (Functional), the ContentsContext (user/system interactions), the LifeCycleSpan (persistent) and the Purpose (descriptive). As another example of chunk descriptor, Fig. 10 presents the descriptor associated to the informal Kyng's scenario based approach whose chunk was sketched in Fig. 5.
Fig. 10. Descriptor of the Kyng's chunk 4. U s i n g S G M L to I m p l e m e n t a n d Q u e r y the Scenario M e t h o d Base SGML (Standard Generalised Markup Language) [Goldfarb90] is an intemational standard language to describe a document using a set of mark ups defined in a grammar. SGML documents are structured as trees. We found the language adequate for representing our scenario method base. Besides SgmlQL [Lemaitre95] is available to query an SGML base of documents. We sketch in this section the SGML structure of our implemented scenario method base and will sketch the query process in the next section.
4.1 Overview of the SGML Structure of the Scenario Method Base The structure of the scenario method base is described in a DTD (Data Type DeEmition). The whole DTD is a document type. It is composed of elements which are characterised by a mark up identifier, constraints on the existence of the opening and closing tags and the definition of their structure, i.e. the component elements. An element can recursively be composed of other elements. Based on this principle, the scenario method base is represented as a document type named METHODBASE (see
205
Fig. 11) which is composed of the element named DESCRIPTIONS. This element consists in a set of descriptions which are themselves called DESCRIPTION. Thus, the structure of the element DESCRIPTIONS is represented by DESCRIPTION* where the <<*))means many. < l
DOCTYPE
METHODBASE
l
ELEMENT
DESCRIPTIONS
-
-
(DESCRIPTION*)
DESCRIPTION
-
-
(META
<
< ! ELEMENT
>
KNOWLEDGE_LEVEL, KNOWKEDGE_LEVEL
< ! <
ELEMENT
! ELEMENT
META
KNOWLEDGE
KNOWLEDGE_LEVEL-
9 ~
LEVEL
-
-
-
)>
(DESCRIPTOR)>
(CHUNKAPPROACH)>
>
Fig. 11. Overview of the SGML structure of the scenario base DESCRIPTIONS
APPLICATION \ 9=~ / \ Graphical_ ~ rw~T,Aar~ - \ v~a~ , c , r , ~ r = v Representation I .......... DESIGN_ TARGET. 9 " . . . . . . . 1"- Informal |
ACTIVITY.." Role9"
Type
" B~DY
MAWR ExamWIN"AOE
" Descriptio CONrF_Xr_ ~ s ~
INTENTION /
/
/
/ ~ G.IDCLI~
l
' -
PRODUCT :
Product .../~CC)NITEXT r ~ r ~ T I O N / 4 x : Scenario Characteristic / ,~v~.~.~,_~ / x Nathe 7 J / ~ LE~ ~ Graphical CONTEXT S~TUATION ~ / SIMPLE INFORMAL_ Represen~tion
~ \ VERB TARGET MANNER / RECOMMEN-Informal_ PRODUCT PART* ~ ~ DATION Description SITUATION_ ~ / ~ Example DESCRIPTION? AC~-ON COMF6SITION R~FINEMENT _LINK LINK LINK ACTION
CONTEXT
CONTEXT
ARGUMENT.*
Fig. 12.Overview of the SGML structure of the scenario method base This way of modelling is recursively applied to integrate all the elements composing the document type. Thus, the DESCRIPTION element is characterised by a metaknowledge level (META_KNOWLEDGE_LEVEL) and a knowledge level (KNOWLEDGE_LEVEL) which are, as presented in the previous sections, respectively composed of a descriptor (DESCRIPTOR), and of either a chunk or an approach denoted by (CHUNK I APPROACH). The resulting structure of the METHODBASE document type is the tree presented in Fig. 12.
206
It is possible to attach attributes to elements to characterise them. For example, the attribute TYPE attached to the element CHUNK characterises its type (FORMAL, INFORMAL). An attribute has a name, a type (enumerated or not) and may be optional (#REQUIRED, #IMPLIED). Underneath is the Sgml description of the mandatory attribute TYPE of CHUNK. <., Arrr, z s r
ChUm( TYPE
(FOat~r., I zz~'oag_m.,)
#REQUIRED'>
The overall description of the document type defined for the scenario base is provided in the appendix 2. In the following section, we illustrate the Sgml document contents with the Jacobson's chunk and its associated descriptor. 4.2 Examples of Sgml Chunks and Chunk Descriptors Let us start with chunk descriptors. As explained in section 3, a chunk descriptor has a situation part and an intention part. According to our approach, the situation pan (see DESCRIPTOR_SITUATION in Fig. 13) is characterised by an application domain (APPLICATION_DOMAIN) and by a design activity (DESIGN_ACTIVITY) which explain respectively, the area and the design activity in which the chunk can be reused. In our example, the area is Object Oriented Applications and the activity is Analysis. <APPLICATION_DOMAIN>Object oriented applications anal ysi s< ~DESIGN_ACTIVITY>
Fig. 13. Example of descriptor situation The intention pan of the descriptor (DESCRIPTOR INTENTION) is illustrated in Fig. 14. It is composed of: * a verb (VERB), to Capture in our example, 9 a target (TARGET) which can either play the role of a result or of an object. This information is denoted in the attribute role of the target by the values ~
207
intention. In Fig. 14, <<Jacobson's Refmement Strategy)) is a simple manner whereas < VERB> Cap t u r e < / V E R B > u s e r interactions < VERB>Defining Use Case ModeI <SIMPLE MANNER> by Jacobson' s Refinement Strategy < ~SIMPLE_MANNER>< ~COMPLEX_MANNER>< ~DESCRIPTOR--INTENTION>
Fig. 14. Example of Sgml descriptor intention Now that we are aware of the Sgml description of the meta-knowledge level of the scenario method base, let's concentrate on the knowledge level. The knowledge level (KNOWLEDGE_LEVEL) (see Fig. 12) is represented in the Sgml structure either by a chunk element (CHUNK) or by an approach (APPROACH) which is composed of chunks (CHUNKS*). As illustrated in Fig. 15, the chunk (CHUNK) element contains two parts: an interface (INTERFACE) and a body (BODY). It has general characteristics, namely a name, a type (formal, informal), an informal description and a reference to a graphical representation.
~Define
Use
Case
Model
by
Jacobson" s
Refinement
type = ~ f o r m a l ~ informal description = ~Defining a use c a s e m o d e l b y J a c o b s o n ' s r e f i n e m e n t s t r a t e g y c o n s i s t s in i d e n t i f y i n g actors, then in c o n s t r u c t i n g u s e c a s e s o u t o f this a c t o r s a n d f i n a l l y in r e f i n i n g the u s e c a s e s c o n s t r u c t e d b e f o r e ~ > < G R A P H I C A L _ R E P R E S E N T A T I O N > < A H R E F = ~ J a c o b P r o c l .g i f ~>< /A> < INTERFACE> < CHUNK SITUATION> Problem S ta t e m e n t < / P R O D U C T _ P A R T >
208
<SITUATION_DESCRIPTION>The p r o b l e m s t a t e m e n t is a n initial textual and informal description of the e x p e c t a t i o n s about the f u t u r e system. It contains some r e q u i r e m e n t s and constraints resulting from interviews with the end users < ~SITUATION_DESCRIPTION> < ~CHUNK_SITUATION> < CHUNK INTENTION> < VERB>Define < T A R G E T r o l e = ~ r e s u l t~ type = ~ s c e n a r i o - b a s e d ~ > U s e Case Model <SIMPLE_MANNER>by Jacobson's Refinement Strategy < ~SIMPLE_MANNER> < ~CHUNK--INTENTION> < B O D Y > < P R O D U C T n a m e = ~Use c a s e m o d e l p r o d u c t ~ informal d e s c r i p t i o n = ~ < ~COMPOSITION_LINK>< ~LINK> < ~COMPOSITION_LINK> < / G U I D E L I N E > < /BODY>< / C H U N K >
Fig. 15. Example of Sgml description of a chunk The interface is composed of two parts : a situation (CHUNK_SITUATION) and an intention (CHUNK_INTENTION). The situation of the chunk interface (CHUNK_SITUATION) is composed of two elements: 9 one or several product parts referenced by PRODUCT_PART* in the Sgml tree, and 9 a description (SITUATION_DESCRIPTION) which is optional. All these elements are strings (#PCDATA).
209
The intention of the chunk (CHUNK_INTENTION) is composed of a verb (VERB), a target (TARGET) and a simple manner (SIMPLE_MANNER). This is exemplified in Fig. 15. Following our definitions in section 2, the body is composed of two parts, the product and the guideline. 9 The product (PRODUCT) is characterised by a name, an informal description, an example of instantiation of the product and a reference to a graphical representation which is a picture stored in the Sgml document. This graphical representation is referenced in Fig. 15 by JacobProd.gif and is presented in Fig. 16. composed-of 1,N
Actor
/\
se o a l I Actor
1,N
I
Use Case Model
~_of
Topic Description
Actor
I
1,N
executes
11
~_
0,N ~
supp~
///
Use Case
II
", us~Case I /
/ Use Case
&
/
I
Use Case
~
~,,extends
~Extension I
x'~ usecase I
II ' e vol Use Case
uses
Fig. 16. JacobProd.gif The guideline (GUIDELINE) can be either represented by an informal description (INFORMAL_RECOMMENDATION) or by a set of links (LINK*) depending on whether the chunk is informal or not. In the case of a formal chunk, the guideline has the form of either a context or a tree of contexts. It is represented in the Sgml structure by the set of links, connecting the contexts one with the others in the tree. Depending on the type of its source context, a link can be either a composition, a refinement or an action link (Fig. 15). The tree structure can be visualised through the graphical representation (GRAPHICAL_REPRESENTATION) element of the structure. This graphical representation is referenced in Fig. 15 by JacobProcl.gif and is presented in Fig. 17.
210
<(Problem Statement), Define (Use Case Model) Res (by Jacobson's refinement strategy) Man >
i
i
<(Pb. St.), Identify (Actor) Res>*
<(Pb. St., Acre0,
Oe e O,e a,e)Re,>.
<({Use Case)}
/ < (Pb. St.), / Identify (Secondary Actor ) Res>* < (Pb. St.), Identify (Primary Actor ) Res>*
<({Use Case}), <(Abstract Use Case), Generalise ({Use Case}) Obj Specialise (Abstract Use Case) Ob into (Abstract Use Case) Res> into (Concrete Use Case) Res> <(Use Case), I Complete (Use Case) Obj < (Pb. St., Actor), < (Pb. St., Actor, UCTOPic), with(Use Case Extension ) Res > I Identify (Use Case Topic) R e s t ( U s e Case) Res>
~ < (Pb. St.,IUse Case), < (Pb. St., Actor, UCTopic), Identify (Extension Use Case Topic) Res> Construct (Concrete Use Case) Res> < (Pb. St., Actor, {UCTOPic}), <(Pb. St. Extension UCTopic), Construct (Abstract yse Case) Res > Elaborate (Extension Use Case Description) Res> <(Pb. St., Actor, {UCTopic}), <(Pb. St., Abstract UCTOPic), Identify (Abstract Use Case Topic) Res> Elaborate (Abstract Use Case Description) Res>
I
I
< (Pb. St., Actor, UCTopic), <(Pb. St., Basic Use Case), Elaborate (Basic Use Case Description) Res> Elaborate (Alternative Use Case Description) Res>
Fig. 17. JacobProcl.gif 5. Examples of SgmlQL Queries This section illustrates the way SgmlQL queries can support the process of reusing scenario chunks. This illustration is based on a reuse scenario that can be described in the following way. Let's assume that Mr Bean,. a requirements engineer is in charge of a project for developing an object oriented application. In the early phase of the project the project used the Enterprise Modelling approach Bubenko94 to model the rich picture of the organisational context in which the system will operate. The result of this step is a set of high level goals. Mr Bean is now faced to the operationalisation of these goals i.e, the specification of the system functions which fulfil these goals. The project constraint is that the system functional specification should be done in an object oriented manner. Mr Bean's belief is that the project should benefit from using a scenario based approach. The argument in favour of such an approach is twofold : (1) the system is too large and complex to be tackled in a global way and, (2) the domain experts are not familiar with semi-formal or formal notations and prefer to communicate with requirements engineers in an informal manner. Thus, we propose to help Mr Bean by querying the Sgml scenario method base to retrieve chunks that match his requirements i.e. chunks which help bridging the gap between goal models and object oriented specifications. We first propose to extract from the method base all the chunks whose situation is based on goals. The query is formulated as follows:
211
Q 1: Select the chunks having goals as product parts of their situation. select text($a->NAME) from $d in every DESCRIPTION within $myfile3, $a in every APPROACH within $d, $pp in first PRODUCTPART within Sa where text($pp) match <>; The answer to this query proposes two chunks, the: 9 Holbrook's, and 9 Cockburn's ones. Mr Bean is not familiar with the Cockbum's approach, but he heard of the Holbrook's one and wish to explore further the possibilities offered by this approach. As the constraint is to produce an object oriented specification, the question is to identify which is the output of the Holbrook's chunk. Q2 gives the answer to this question. Q2: Select the target generated by the <>chunk. select text(first TARGET within $cm) from $d in every DESCRIPTION within $myfile, $descro in every DESCRIPTOR within $d, $cm in every COMPLEXMANNER within $descro, $a in every APPROACH within Sd where text($a->name) match <>; The target is: Use Scenario An access to the PRODUCT part of the chunk in the Sgml document (to the HolbProd.gif in particular) convinces Mr Bean that unfortunately, the output of the Hoolbrook's chunk is not an object oriented specification. Thus, we suggest to search for another scenario based chunk that supports the transformation of input scenarios into object-oriented models. This can be done with query Q3, presented below. Q3: Select chunks which are using scenario or scenario-based product as input and generate an analysis model as output. select text($c->NAME) from $d in every DESCRIPTION within $myffle, $descro in every DESCRIPTOR within $d, $c in every CHUNK within $d, $pp in first PRODUCTPART within $c where ((text($pp) match <<scenario>>) or (text($pp) match <(use-case>>)) and (text(first TARGET within $descro) match <); This query results in the selection of the chunk named <>.Mr Bean is happy and decides to explore further on the solution consisting of combining
3 A preliminary query (global $myfile =file a MethodBase.sgml 0 should be typed in order to indicates which sgml document (here MethodBase.sgml) is queried.
212
the Holbrook's chunk based on use scenarios with the Define analysis model chunk supporting construction of an analysis model out of scenarios. Even short and schematic, this example illustrates the use of SgmlQL queries for retrieving scenario base chunks meeting the requirements of a user. It suggests at least two comments: (1) the need for a thesaurus to support an efficient query process and (2) the possibility to capitalise from experience, for instance, by inserting in the method base the new approach when fully generated. This insertion can be done using specific commands provided by SgmlQ1.
6. Conclusion In this paper we propose an approach for supporting the reuse of scenario based chunks made available in a scenario method base. The motivation for developing such an approach was twofold: first, there exists a large corpus of available scenariobased approaches that has not been formalised yet, and secondly there is an explicit need for incorporating scenario-based approaches in existing methods. FUSION, OMT and UML are examples of such enhancements that the method engineers would like to reproduce. The proposed approach advocates a modular representation of scenario chunks and an intentional description of their reuse context. The former results is cohesive chunks which are applicable in specific situations for specific purposes whereas the latter provides contextual information identifying in which specific design situations for which specific design intentions the chunks are reusable. The paper also reports on the implementation of a scenario method base in SGML and illustrates the reuse process through an example. Future work shall concentrate on developing guidelines to integrate scenario chunks in existing methods and the implementation of these guidelines in the Sgml context. Besides, in order to support the process for retrieving chunks matching specific requirements we are developing a set of SgmlQL macro-queries. Finally we shall work on an HTML presentation of the query responses.
7. References Booch97: G. Booch, J. Rumbaugh, I. Jacobson, <, Proc. of the First International Conference on Requirements Engineering, April 1994, Colorado Springs, Colorado. Carroll95 J.M. Caroll, ~ The Scenario Perspective on System Development >), in J.M. Carroll (ed.), Scenario- Based Design: Envisioning Work and Technology in System Development (1995).
213
Cockburn95 A. Cockburn, ~Structuring use cases with goals)), Technical report, Human and Technology, 7691 Dell Rd, Salt Lake City, UT 84121, HaT.TR.95.1, http://members.aol.com/acocbum/papers/usecases.htm (1995). Coleman97: D. Coleman, P. Arnold, S. Bodoff, C. Dollin, H. Gilchrist, F. Hayes, P. Jeremaes, ~Object-Oriented Development: The FUSION Method )), Prentice Hall, 1994. De Antonellis91: V. De-Antonellis., B. Pemici, P. Samarati, (1991) ~F-ORM METHOD: A Methodology for Reusing Specificatiom), In Object Oriented Approach in Iforamtion Systems, Van Assche F., Moulin b., Rolland C. (eds), North Holland, 1991 Erickson95: T. Erickson, ~Notes on Design Practices: Stories and Prototypes as Catalysts for Communication)), in J.M. Carroll (ed.), Scenario- Based Design: Envisioning Work and Technology in System Development (1995). Filippidou97: D. Filippidou, P. Loucopoulos, <<Using Scenarios to Validate Requirements in a Plausibility-Centred Approach)), Proc. Of the 9Th Conference on Advanced Information Systems Engineering, Barcelona, Catalonia, Spain, June 1997. Glinz95 M. Glinz, ~An Integrated Formal Model of Scenario based on Statecharts)~, Lecture Notes in Computer Science'95,pages 254-271, 1995. Goldfarb90: 1990.
C. F. Goldfarb, ~ The SGML Handbook )), Oxford Clarendon Press,
Holbrook90 C. H. Holbrook III, ~ A Scenario-Based Methodology for Conducting Requirement Elicitation)), ACM SIGSOFT Software Engineering Notes, 15(1), pp.95104, 1990. Hsia94 Hsia P, Samuel J, Gao J, D., Toyoshima, Y. and Chen ,C. (1994) ~Formal Approach to Scenario Analysis)), IEEE Software, 11, 33-41 Jacobson92 I. Jacobson, M. Christerson, P. Jonsson and G. Oevergaard, ~Object Oriented Software Engineering: a Use Case Driven Approach)), (Addison-Wesley, 1992). Jacobson95a I. Jacobson, ~The Use Case Construct in Object-Oriented Software Engineering)), in John M. Carroll (ed.), Scenario-Based Design: Envisioning Work and Technology in System Development (John Wiley and Sons, 1995) 309-336. Jacobson95b I. Jacobson, M. Ericsson and A. Jacobson, ~The Object Advantage, Business Process Reengineering with Object Technology)) (Addison-Wesley Publishing Company, 1995). Jarke97 : M. Jarke, K. Pohl, P. Haumer, K. Weidenhaupt, E. Dubois, P. Heymans, C. Rolland, C. Ben Achour, C. Cauvet, J. Ralyte, A. Sutcliffe, N. A. Maiden and S. Minocha, ~Scenario use in european software organisations - Results from site visits and Questionnaires)), Esprit Reactive Long Term Research Project, 21.903 CREWS, Deliverable Wl : Industrial Problem Capture Working Group, 1997.
214
Johnson95 P. Johnson, H. Johnson and S. Wilson, ~Rapid Prototyping of User Interfaces driven by Task Models)), in John M. Carroll (ed.), Scenario-Based Design: Envisioning Work and Technology in System Development (John Wiley and Sons, 1995) 209-246. Karat95: J. Karat, <~ScenarioUse in the Design of a Speech Recognition Systenv), in J.M. Carroll (ed.), Scenario- Based Design: Envisioning Work and Technology in System Development (1995). Kyng95: M. Kyng, Creating Contexts for Design, in John M. Carroll (ed.), ~Scenario-Based Design: Envisioning Work and Technology in System Developmenb~ (John Wiley and Sons, 1995) 85-107. Lalioti95: V. Lalioti and B.Theodoulidis, ~Use of Scenarios for Validation of Conceptual Specificatiom~, Proceedings of the Sixth Workshop on the Next Generation of CASE Tools, Jyvaskyla, Finland, June 1995. Leite97: J.C.S. do Prado Leite, G. Rossi, F. Balaguer, A. Maiorana, G. Kaplan, G. Hadad and A. Oliveros, ~Enhancing a Requirements Baseline with Scenarios~, In Third IEEE International Symposium On Requirements Engineering (RE'97), Antapolis, Maryland (IEEE Computer Society Press, 1997) 44-53. Lemaitre 95: J. Lemaitre, E. Murisasco, M. Rolbert, SgmlQL, ~Un langage d'interrogation de documents SGML~, Proceedings of the l lth conference on Advanced DataBases, August 1995, Nancy, France. Nielsen95: J. Nielsen, Scenarios in Discount Usability Engineering, in John M. Carroll (ed.), ~Scenario-Based Design: Envisioning Work and Technology in System Development~ (John Wiley and Sons, 1995) 59-85. Plihon95: V. Plihon, C. RoUand, ~Modelling Ways-of-Working~ Proc. Of the 7th Int. Conf. On ~(Advanced Information Systems Engineering~, (CAISE), Springer Verlag (Pub.), 1995. Potts94: C. Potts, K. Takahashi and A.I. Anton, ~dnquiry-based Requirements Analysis~, in IEEE Software 11(2) (1994) 21-32. Prat97: N. Prat, ~Goal Formalisation and Classification for Requirements Engineering~, Proceedings of the Third International Workshop on Requirements Engineering: Foundations of Software Quality REFSQ'9, Barcelona, june 1997. Prieto-Diaz87: R. Prieto-Diaz, P. Freeman, ~ Classifying Software for Reusability~, IEEE Software, Vol 4 No 1, 1987. Regnel195: B. Regnell, K. Kimbler and A. Wesslen, ~dmproving the Use Case Driven Approach to Requirements Engineering~, in the Second IEEE International Symposium On Requirements Engineering, York, England (I.C.S. Press, March 1995) 40-47. Roberston95: S.P. Robertson, ~Generating Object-Oriented Design Representations via Scenarios Queries~, in John M. Carroll (ed.), Scenario-Based Design: Envisioning
215
Work and Technology in System Development (John Wiley and Sons, 1995) 279308. Rolland94: C. RoUand, G. Grosz, ~(A General Framework for Describing the Requirements Engineering Process~, IEEE Conference on Systems Man and Cybernetics, CSMC94, San Antonio, Texas, 1994. Rolland96: C. Rolland, N. Prakash, ~A proposal for Context-Specific Method Engineering~, IFIP TC8 Working Conference on Method Engineering, Atlanta, Gerorgie, USA, 1996 Rolland97: C. Rolland, C. Ben Achour, C. Cauvet, J. Ralyte, A. Sutcliffe, N.A.M. Maiden, M. Jarke, P. Haumer, K. Pohl, E. Dubois and P. Heymans, ~A Proposal for a Scenario Classification Framework~, ESPRIT Reactive Long Term Research Project 21.903 CREWS, Deliverable I 1: Initial Integration Workpackage (1997). Rumbaugh91: J. Rumbaugh, M. Blaha, W. Premerlani, F. Eddy, and W. Lorensen, ~tObject-Oriented Modeling and Desigm), (Prentice Hall, 1991). Rumbaugh94: J. Rumbaugh, ~Getting started, using use cases to capture requirements)), Journal of Object Oriented Programming, Sept. 1994. Rumbaugh96: J. Rumbaugh and G. Booch, ~Unified Method)), Notation Summary Version 0.8 (Rational Software Corporation, 1996). Rubin95: K.S. Rubin and A. Goldberg, ~Object Behaviour Analysis)), Communications of the ACM 35(9) (1992) 48-62. Thom~93: B. Thom6, ~Systems Engineering : Principles and Practice of Computerbased Systems Engineering)), in B. Thom6 (ed), John Wiley & Sons (1993). Wirfs-Brock95: R. Wirfs-Brock, ~Designing Objects and their Interactions: A Brief Look at Responsability-driven Design)), in John M. Carroll (ed.), ~Scenario-Based Design: Envisioning Work and Technology in System Development~) (John Wiley and Sons, 1995) 337-360. Young87: M. R. Young, P. B. Barnard, ~The Use of Scenario in Human-Computer Interaction Research: Turbocharging the tortoise of Cumulative Science)), CHI + GI 87 Human Factors in Computing Systems and Graphics Interface, Toronto, 1987. 8. A p p e n d i c e s 8.1 Appendix 1 : Charaeterising Scenarios
In Rolland97 a framework for scenario classification is proposed to characterise scenarios according to a certain number of facets which are grouped into views. Four views have been identified : 1. the form view, 2. the contents view, 3. the purpose view and 4. the life cycle view.
216
The form view answers the question 'in which form is a scenario expressed ?'. The response is provided through two facets namely the description facet and the presentation facet. 9 The description facet characterises the level of formality and the medium used for the scenario description. Texts, graphics, images, videos and software prototyping are examples of media. Note that several media can be used at the same time for describing a scenario. 9 The presentation facet tells whether a scenario is static or animated, and its interactivity i.e. the capabilities offered to the user to control the way the scenario progresses through time. Consequently, in the descriptor the form view is represented by four properties of scenarios namely, FormDescriptionMedium, FormDescriptionNotation, FormPresentationAnimation and FormPresentation Interactivity. The contents view answers the question 'what is the knowledge expressed in a scenario ?'. The response is provided through four facets namely the abstraction facet, the context facet, the argumentation facet and the coverage facet. 9 The abstraction facet indicates whether the scenario is concrete, abstract or mixed. 9 The context facet explains in which kind of context the scenarios are used. System internal, system interaction, organisational context and organisational environment are examples of contexts where the scenarios can be used. 9 The argumentation facet indicates whether argumentation concepts are used within the scenarios or not. 9 The coverage facet indicates the kind of information captured in the scenarios, i.e. whether it is functional, non fimctional or intentional. The information concerning the structure, the function and the behaviour are qualified as functional, the ones which are tackling performance, time constraints, cost constraint, user support, documentation examples, back up/recovery, maintainability, flexibility, portability, security/safety, design constraints, error handling are non functional and the information concerning goal, problem, responsibility, opportunity cause and goal dependency are said intentional. This leads to the following properties of scenarios in the chunk descriptor: ContentsAbstraction, ContentsContext, ContentsArgumentation and ContentsCoverage. The purpose view answers the question 'why using a scenario ?'. The response is provided through three criteria. A scenario can be used 9 in a descriptive purpose, i.e. for describing something which happens in the real world,
217
9 9
in a exploratory purpose i.e. for constructing requirements elicitations, or in a explanatory purpose, i.e. when explanations about the rationale of these issues are required.
The purpose perspective is associated in the chunk descriptor to the property called Purpose. The life cycle view is characterised by two facets: the life span facet and the operation facet. It explains 'how to manipulate scenarios'. 9 The life span facet indicates whether the scenario is transient or persistent in the RE process. 9 The operation facet defines if and how scenarios are captured, refined, integrated, expanded, and deleted. This is represented in the chunk descriptor by two properties, namely LifeCycleSpan and LifeCycleOperation. More details on the scenario classification framework and its application on several scenario based approaches can be found in Rolland 97. 8.2 Appendix 2 : The Scenario Method Base Structure < ! DOCTYPE <
!
ELEMENT
METHODBASE DESCRIPTIONS
< l ELEMENT
DESCRIPTION
< l ELEMENT
META
-
-
(DESCRIPTION*)
9
(META_KNOWLEDGE_LEVEL, KNOWLEDGE_LEVEL)
<
l ELEMENT
KNOWLEDGE
LEVEL
DESCRIPTOR (DESCRIPTOR_SITUATION, DESCRIPTOR
SITUATION
< ! ELEMENT
APPLICATION
DOMAIN
< ! ELEMENT
DESIGN
< ! ELEMENT
DESCRIPTOR
<
!
ELEMENT
-
DESCRIPTOR_INTENTION)
ACTIVITY
9
-
(APPLICATION_DOMAIN, -
-
-
-
INTENTION
DESIGN_ACTIVITY)
(#PCDATA)
9
(#PCDATA)
9
(VERB,
ELEMENT
< 1 ELEMENT
(VERB
I TARGET)
COMPLEX
-
-
MANNER
9
TARGET, COMPLEX_MANNER)
<1
9
(DESCRIPTOR) 9
(#PCDATA) (VERB,
>
9
TARGET,
w
SIMPLE_MANNER) < ! ELEMENT
SIMPLE
< ! ELEMENT
KNOWLEDGE
< 2 ELEMENT
CHUNK
MANNER LEVEL
-
-
(#PCDATA)
-
-
(CHUNK
-
-
(INTERFACE,
IAPPROACH)> BODY,
GRAPHICAL_REPRESENTATION) -
-
< ! ELEMENT
INTERFACE
< ! ELEMENT
CONTEXT
SITUATION
-
-
(PRODUCT_PART*,
< ! ELEMENT
PRODUCT
PART
-
-
(# P C D A T A )
< ! ELEMENT
SITUATION
-
-
(#PCDATA)
< ! ELEMENT
CONTEXT
-
-
(VERB,
< ! ELEMENT
BODY
>
(CONTEXT_
SITUATION,
CONTEXT_INTENTION)
>
SITUATION_DESCRIPTION?) DESCRIPTION INTENTION
>
>
9
> 9
TARGET, SIMPLE_MANNER)
(PRODUCT,
GUIDELINE)
9
9
218
< ! ELEMENT
PRODUCT
-
-
< ! ELEMENT
GRAPHICAL
< ! ELEMENT
GUIDELINE
ELEMENT < ! ELEMENT
INFORMAL LINK
< !
REFINEIKENT_ LINK, ACTION_LINK)> COMPOSITION LINK - (#PCDATA) > REFINEMENT LINK - (#PCDATA,ARGUMENT?) 9 ARGUMENT - (#PCDATA)> ACTION LINK - (ACTION) 9 ACTION - (#PCDATA) > APPROACH - (CHUNK*) 9 PRODUCT NAME (#PCDATA) #REQUIRED> PRODUCT INFORMAL DESCRIPTION
(GRAPHICAL_REPRESENTATION) - (#PCDATA) 9
REPRESENTATION-
-
>
(INFORMAL_RECOMMENDATION I LINK*) 9
< !
ELEMENT ELEMENT ELEMENT ELEMENT ELEMENT ELEMENT ATTLIST ATTLIST
RECOMMENDATION-
-
(#PCDATA) 9 (COMPOSITION_LINK
I
(# P C D A T A ) #REQUIRED> EXAMPLE (#PCDATA) #IMPLIED> (OBJECT I RESULT) #IMPLIED 9 (SCENARIO-BASED I NON SCENARIO_BASED) #IMPLIED> FORMDESCRIPTIONMEDIUM (#PCDATA) #IMPLIED 9 FORMDESCRIPTIONNOTATION (# P C D A T A ) #IMPLIED 9 FORMPRESENTATIONANIMATION (# P C D A T A ) # I M P L I E D > FORMPRESENTATIONINTERACTIVITY (# P C D A T A ) #IMPLIED> CONTENTSABSTRACTION (#PCDATA) #IMPLIED 9 CONTENTSCONTEXT (# P C D A T A ) #IMPLIED> CONTENTSARGUMENTATION (#PCDATA) #IMPLIED> CONTENTSCOVERAGE (#PCDATA) #IMPLIED 9 LIFECYCLESPAN (#PCDATA) #IMPLIED 9 LIFECYCLEOPERATION (#PCDATA) #IMPLIED> PURPOSE (#PCDATA) #IMPLIED 9 SRC (#PCDATA) #REQUIRED 9 NAME (# P C D A T A ) #REQUIRED> TYPE (FORMAL I INFORMAL) #REQUIRED 9 INFORMAL DESCRIPTION (#PCDATA) #REQUIRED> EXAMPLE (#PCDATA) #IMPLIED 9
Change Analysis and Management in a Reuse-Oriented Software Development Setting Wing Lam Department of Computer Science, University of Hertfordshire College Lane, Hatfield, Herts ALl0 9AB, UK [email protected], Phone: +44 (0)1707 284337, Fax: +44 (0)1707 284303
Abstract. This paper discusses
the need for systematic and methodical approaches for change analysis and management in software projects. We are not concerned with specific techniques (such as program slicing or dependency graphs), but with the overall project-level process of handling change, which we feel is underrepresented in the current literature. The paper describes the efforts being made to manage change at Software Development Services (SDS), a small commercial organisation that specialises in the development of Customer Complaint Systems (CCSs). Because SDS recognises that most change is specific to their domain (CCSs), we are taking a domain-specific approach to change analysis and management. Our main contribution in this paper is a framework for the change process, called the Change Cycle, which is concerned with identifying and formalising reusable change knowledge. The paper reviews current methods and techniques for handling change, and discusses the desirable characteristics of a change process. We then describe the Change Cycle in detail and our experience of using it at SDS. Tool support for change is also outlined, as is our approach to evaluating this work. We conclude that change analysis and management should be treated as an integral part of the software process, not as an adjunct to it.
1. The Need for Systematic Approaches for Handling Change Software systems are not static, but evolve. Fluctuating user requirements, commercial pressures, organisational transition and demands for interoperability (e.g. the Internet) all contribute to volatility in the software process 14, 18. As 16 noted,
"software systems must change and adapt to the environment, or become progressively less useful". Increasingly then, today's software engineers need systematic approaches for dealing with change. This paper describes the efforts being made to manage change at Software Development Services (SDS), a small commercial organisation than specialises in the development of Customer Complaint Systems (CCSs). Because CCSs are similar in functionality, SDS has adopted a reuse-oriented software process, where individual CCSs are tailored from a generic CCS prototype. SDS has observed that many of the kinds of changes made to one CCS are similar to the kinds of changes made to other CCSs, i.e. much change is domain-specific. This paper describes how we attempted to exploit this observation in developing a systematic approach to the analysis and management of change at SDS. The format for this paper is as follows. The next section, Section 2, describes the problem of change from the perspective of SDS. Section 3 reviews current methods
220
for handling change in the literature and indicates how the approach discussed here differs. Section 4 discusses the desirable characteristics of a change process. Section 5 presents our framework for change analysis which we call the 'change cycle'. Section 6 describes the application of the change cycle to change problems at SDS. Section 7 outlines tool support for change and Section 8 the strategy we are using to evaluate our work. Finally, Section 9 concludes.
2. Change in a Reuse-Oriented Software Process SDS was formed in 1994 to meet the growing market for Customer Complaint Systems (CCSs). The majority of clients for CCSs are retail outlets. In short, a CCS records individual customer complaints in a desired format, tracks the complaint from initiation through to resolution and performs a variety of anlayses over complaint data (e.g. % of complaints resolved within a given time, distribution of complaints relating to a defined service or product group). Like many commercial applications today, CCSs are typically required to run under the Windows environment, either as a standalone or networked (multi-user) application. Because CCSs tend to share similar functionality, SDS has adopted a reuseoriented software process, where an individual CCS for a customer is tailored from a generic CCS prototype. In the SDS software process, change is seen as a central part of the software process of which there are three main categories (Figure 1).
Figure 1 Types of change 1. Tailoring changes (T-Changes). This is where SDS identifies with the customer the changes needed to tailor the generic CCS to their specific CCS requirements.
2. Requirement changes (R-Changes). In addition, like many software systems, 3.
changes (which are customer-driven) also occurs during the development process (e.g. demonstrations, trials) and after a CCS has been installed on-site. Evolutionary changes (E-Changes). The generic CCS is itself the subject of evolutionary improvements. As SDS develops more CCSs, we expect the generic CCS to gradually become more 'mature'.
221
This paper is concerned primarily with R-Change, which 9 identify as a major challenge to software engineering (use of the word 'change' from this point on will refer to R-change). Like many 'real world' applications, changing and evolving requirements is a prominent feature of CCS software development projects. As CCSs are highly interactive systems, SDS use a user-centred and prototyping approach to their development. While such an approach is useful in 'discovering' customer requirements, we have found that there is little or no control over the change process itself. SDS has observed that the kinds of changes that occur in the development of one CCS are often repeated in the development of other CCSs, i.e. change is domainspecific. As such, one of the goals of SDS is to acquire an understanding of these 'patterns' of change. It is envisaged that achieving this will facilitate the development of future CCSs, in particular, helping SDS to develop capabilities in the following areas:
1. 2. 3.
Change anticipation. Anticipate change (e.g. to expect the introduction of new requirements) in order to plan ahead and address change in a pro-active rather than reactive manner. Change categorisation and change strategy reuse. Identify and classify different types of change in order to formulate and re-use strategies for coping with change. Change estimation. Collect cost and time-scale data for types of change in order to aid estimation (e.g. time and cost to implement a change).
These capabilities, however, depend on the capture and reuse of domain-specific change knowledge, which is an aspect of change that is not addressed by the current literature.
3. Current Methods and Techniques for Handling Change There are a number of distinct research areas that provide an angle for addressing aspects of software evolution, some of which are particularly relevant to change at the requirements level.
Process models for software evolution attempt to define high-level process models that attempt to either reduce or eradicate the perceived gap between software development and maintenance. The IEEE software maintenance model, described by 3, proposes a seven-step 'waterfall-like' process: a) problem identification, b) analysis, c) design, d) implementation, e) system test, f) acceptance test and g) delivery. This, however, provides no real detailed guidance for the software maintainer. The FEAST (Feedback and Software Technology) model 16 is an advancement of Lehman's well-known E-process model. In the FEAST model, the software process is modelled in a dynamic fashion as a continuous system varying over time. Execution of the process is through control loops, which produce 'feedback' to drive the next steps in the process. However, FEAST is a theoretical (rather than practical) model, and questions about the
222
9
9
9
9
9
level of modelling (fine-grain modelling is likely to result in extremely complicated models) and nature of control loops (open or closed) are unclear. Heuristic support includes guidelines for managing change, which might be appropriate at different levels of support, e.g. managerial-level, project-level, and task-level. 22 identify 3 strategies for change: a) identifying change early in the lifecycle, b) facilitating the incorporation of change and c) reducing change. They offer further guidelines about techniques appropriate to each strategy, e.g. the use of prototyping, simulation, work-throughs and scenarios in strategy a). Evolutionary delivery methods 7 encourage incremental development and early feedback. Evolutionary delivery methods do, however, rely on being able to compartmentalise the system and to deliver it in stages. The application of evolutionary delivery methods has generally been used at a macro-level to deal with change from a user-perspective, such as in the form of prototyping. However, support for other aspects of change such as studying the impact of change or change anticipation is not addressed. Logic languages can help to reason formally about change and impact 24, 4. 4 describes a language with goal-structures that capture the relationships between goals and sub-goals, which can be used to reason about alternative design decisions and analyse trade-offs. One problem here is that there are likely to be many influences on the software process, and a language that attempts to capture all these concepts is likely to be both large and complex (even before one attempts to apply it). In addition, there are validation issues as the ability to do any useful reasoning relies on having realistic and representative models of the reallife process. Taceability attempts to define the important associative relations between and across actors and artefacts in the software process 20. Traceability is important in determining how change to one software artefact might affect another artefact. Traceability at the implementation level is supported by techniques such as program slicing 24 and program dependence graphs 19. However, there is an increasing need to extend traceability to earlier levels of software engineering. One issue is the granularity at which traceability is performed. For example, in requirements engineering, traceability might be performed at the requirements document level and/or at the level of individual requirements. Environments have been proposed to support change impact and change propagation. Environments tend to support a mixture of automatic and user-assisted change operations. Such environments operate at the implementation level and use program dependency graphs 8, 11. An expert system environment has also been suggested by 1.
The work so far in this area concentrates on general and non-domain-specific methods and techniques for supporting change. Our work differs, in that we are concerned with domain-specific methods for supporting change in the software process. It is generally recognised that domain knowledge is central to the enactment of many software processes, e.g. requirements engineering 6. Our general hypothesis here is that individual application domains exhibit "patterns" of change (particularly at the requirements level), and that understanding these change patterns can facilitate the evolution of future systems in the domain, as well as provide guidance for the application of non-domain-specific techniques. In short, we are assuming that change in
223
the past (historical change) will be indicative of change in the future. This is not an unusual assumption in software engineering, e.g. historical data forms the basis for reliable estimation models 21 . In the following, we present a view of the change process, called the change cycle, which we have developed in our current work. First, however, we feel it is beneficial to discuss what the general, desirable characteristics of a change process are.
4. Desirable Characteristics of a Change Process A prescriptive process describes the activities necessary to complete a goal and the order in which those activities are to be carried out 25. 17 describes some of the desirable properties of a prescriptive process to be: convey information readily, have reasons supporting their definition, have formally defined syntax and semantics, comprehensive, describe and integrate different levels of abstraction and is reusable. One approach for coping with change is for organisations to develop prescriptive models of the change process. From our discussion in the previous sections, a prescriptive model of the change process should help in some, if not all, of the following ways: 9 9
9
9
Help in change prevention. Not all change can or should be prevented. However, it is sensible to prevent or least minimise unnecessary or 'risky' change. Help in impact analysis. It should be possible to reason, or at least conjecture, about the potential effects of change on software artefacts and the software process. Help in change planning. Change should be planned into the software process rather than being treated as an adjunct to it. Also, the sequencing of change and the collective effect of change over change in isolation should be considered. Help in stability assurance. After a change has been implemented, assurance procedures should exist to ensure that integrity (software and documentation) is maintained. This is particularly important in the case of safety-critical systems.
Ultimately, dealing properly with change at the time will save on re-work further down the software time-line, just as dealing with problems at the requirements level is more cost-effective than dealing with the problem at implementation.
5. The Change Cycle: A Framework for Change Analysis A framework for change analysis that we have developed is the change cycle (Figure 1). The change cycle provides a concise and integrated view of the change process that emphasises the domain-specific and reuse perspectives that are important in our work.
224
Figure 2 The change cycle At the centre of tile change cycle is an evolving body of change knowledge. The change cycle is composed of four distinct sectors which contribute to the change knowledge:
1. 2. 3. 4.
Traceability analysis. An analysis of the dependencies between and across software artefacts and actors in the software process. Change capture. The modelling and capture of change on specific projects. Reuse analysis. The derivation of generic change patterns and reusable change knowledge. Change management. The application of generic change patterns and reusable change knowledge in the management of change on a specific project.
Steps 2-4 share a similar overall process to that of domain-specific reuse approaches, for example, the reuse of domain-specific requirements as described by [12]. We argue that this should not be surprising, as requirements engineering, like change analysis and management, is a process that is strongly guided by domain expertise [6, 5]. The change process in Figure 2 is a cycle because each iteration of the change cycle refines and re-uses the body of change knowledge. We view change knowledge as something which evolves over time and with development experience in the domain. It is both unreasonable and unrealistic to expect a single analysis of change to miricuously produce a complete and definitive knowledge about change, especially in complex real-world domains (as opposed to academically-bounded domains often used in the literature). Each iteration of the change cycle, however, can contribute to an explicitly documented body of change knowledge. One role of the process cycle is to organise specific change analysis and management techniques within the broader change process. Table 1 outlines some of the specific techniques which can be used in each sector of the change cycle. The
225
'Focus of techniques' field in Table 1 describes the general purpose of the set of techniques; the 'Level/scope of usage' indicates the intended scope of these techniques. Our list of techniques is not complete, and refer to the ones which we have used so far in our work. 22 also describe a more general set of techniques which could also be fitted into the context of the change cycle).
Table 1 Techniques in the change cycle
Focus of techniques
Techniques
Level~scope of usage
Traceability analysis
Model associative relationships in the existing software process,
Organisation-specific
Change capture Reuse analysis
Acquisition of change knowledge through the analysis of exemplars. Identification and formalisation of reusable change knowledge,
Change management
Application of reusable change knowledge and the provision of feedback for its improvement.
The grounding for many of these techniques should be familiar to those with an appreciation of basic software engineering methods, e.g. use-case modelling, metrics and traceability. We believe that a systematic process for managing change can be achieved by combining and applying these basic software engineering concepts. We provide an illustration of the use of some of these techniques at SDS in the next section.
6. Application at SDS Because the CCS domain is real-world domain that is subject to real change and real customers, we feel the domain is a suitable domain upon which to concentrate our research efforts on. In addition, the domain is relatively intuitive (compared to, for example, the author's previous work on aircraft control systems 12), so the learning curve for understanding the domain did not adversely impede our research. We have currently performed one iteration (which took about one person-week of effort) of the change cycle to study change in the customer complaints domain at SDS. The result of this iteration is an initial body of change knowledge, which we will refine in future iterations. We intend to perform one iteration per CCS developed at SDS, in order to refine our change knowledge at every opportunity. In the following, we describe the application of specific techniques in each sector of the change cycle. However, instead of burdening the reader with the fine details of our application, we draw out the main goal-task activities in each sector.
226
6.1 Sector 1: Traceability analysis Goal: Study documentation traceability Task: Draw the documentation architecture Just as an implementation has an architecture, so do requirements and designs. The importance of the requirements architecture in requirements reuse is described by [15]. The requirements and design architecture is often reflected in the documentation architecture. The documentation architecture describes the internal structure of system documentation and the dependency relationships that exist between documentation units within the same document and across different documents. At SDS, a CCS requirements document is produced from the generic CCS. From this, a CCS design document is produced. Figure 3 shows the documentation architecture. Here, the small boxes depict the documentation units, and the arrows the dependency relationships between units that we have uniquely identified.
Figure 3 The documentation architecture Goal: Examine the nature of documentation dependencies Task: Instantiate documentation dependency templates We used documentation dependency templates (DDTs) to help examine the nature of dependencies in the documentation architecture, i.e. make clear why there is a dependency not just the fact that there is. An example of an instantiated DDT is shown in Table 2.
227
Table 2 An instantiated DDT Documentation dependency Documentation structural unit 1 Documentation structural unit 2 Dependency Consistency rules
D7 Data-requirements-CCS-requirements-document DB-table-structures-CCS-design-document Requirements to DB-table-structure. 1. Data requirements must match the underlying database table structure. 2. Data requirements must be able to be met through processing from data stored according to the underlying database table structure.
We instantiated DDTs for each dependency identified in the document architecture to understand, at a fine-grain level, the traceability concerns in CCS development at SDS.
6.2 Sector 2: Change capture Goal: Elicit the change process followed by developers for dealing with specific kinds of change Task: Walk-through change scenarios An important element of change capture was to understand how staff as SDS deal with different kinds of change. We elicited processes for dealing with change by identifying and walking-through different change scenarios. We identified common change scenarios based on an examination of the kinds of changes that had occurred in previous CCSs. We used event traces to represent and discuss change scenarios because of their ease of understanding and semi-rigorous approach. Figure 4 shows an example of a change scenario in the case of a change to a screen layout (i.e. visual requirement). CCS Customer
SDS Customer Interface
SDS Development Team
prototype demonstration
i
screen change request i
CCS Requirements Document
sketch u p n e w screen
document
screen change ql~
notificatton of
check amount of work i n v o l v e d - -
check any additional 4 - data implications - -
I 1 ~ imphcat=ons obtain customer agreement q#~ re-desngn s c r e e n - prototype demonstration
Figure 4 A change scenario
update requirements document
228
Goal: Model and capture information about specific changes Task: Characterise the change attributes of a change Change scenarios capture the process of change. To model other features of change, we have identified a set of change attributes, which we formulated during discussion with staff at SDS. W e have classified the change attributes into four categories that reflect the aspects o f change o f most concern to SDS.
1. 2. 3.
Source. Where the change emanates from. Severity. The severity o f the change. Effect. The effect that the change has on software artefacts and the software
4.
Strategy. The strategy (such as process and pitfalls) used to deal with the change
process. on a specific project. Table 3 shows the change attributes (with examples) we have identified so far, but which we acknowledge m a y not be complete.
Table 3 Change attributes Attribute Catesory
Attribute
Example Attribute Values
What Source
Add 'counter' display to the new complaints form. John Smith
Effect
Description Who (made the change request) When Why Time Cost (internal) Change artefacts
Strategy
Process
Severity
Pitfalls
Prototype demonstration, 12-10-96 Counter needed to display number of unresolved complaints 5 person-hours s at s 1. Screen (visual) requirements (CCS Requirements documen0 2. Data requirements (CCS Requirements document) 3. Screen layouts (CCS Design document) 4. New module specification (CCS Design document) 5. New complaints form (code) 6. New function in function library (code) 1. Re-design new screen layout. 2, Design counter function. 3. Implement new screen. 4. Implement new function. 5. Test 1. 2. 3. 4.
Adding a new field may require a significant redesign to the original form. Adding a new field may violate screen design consistencies. New field of this type (counter) should 'match' existing fields of the same type. Counter fields should not add or modify data in the underlying database.
229
Finding the values for change attributes for a particular change takes place over the life of a change, i.e. from the inception of the change through to its completion. For example, the severity of a change in terms of its time and cost can only be known once the change has been completed (though we might have some idea beforehand of its likely severity - - this is an example of the kind of reusable knowledge we are trying to exploit). It should be recognised that what we are doing here is taking a direct modelling approach, i.e. we view change as an explicit concept in the software process just as we view each requirement or each module of code as an explicit concept. Each change in the development of a CCS can be captured in terms of our change attributes. This provides SDS with a convenient way of documenting change, but also enables SDS to build up an empirical and historical record of change upon which to generalise (sector 4 in our change cycle).
6.3 Sector 3: Reuse analysis Goal: Establish a set of reusable change patterns Task: Build-up a change use-case hierarchy Through discussions at SDS over many concrete change scenarios, we were able to identify common change scenarios which we call change use-cases (as in the usecases described by 10). Change use-cases are reusable across CCS projects. Change scenarios are instances of a change use-case. For example, our discussions at SDS quickly revealed three common change use-cases:
1. 2. 3.
Visual-change-use-case. Changes to the screen layout (visual requirements), which are often raised during prototype demonstrations with the customer during an iterative development process. Report-change-use-case. Change to reports, especially when 'example' reports are produced from test data and are given to the customer for inspection. Data-change-use-case. Change to data requirements, which often includes the inclusion of new information to be recorded by the system (customers tend not be know at the start what their exact data requirements are).
We suspect that there are many more change use-cases that we have not yet identified. In particular, there are also specialised forms of the change use cases mentioned already. A report change use-case, for example, may have a report-layout-change-usecase and a new-data-change-use-case. Building up a complete picture of these change use-cases in the customer complaints domain in terms of a change use-case hierarchy (Figure 5), is part of our on-going work.
230
Figure 5 Evolving a change use-case hierarchy We develop a generic event trace for each change use-case by abstracting, using manual inspection methods, from concrete event traces. The change use-case thus captures reusable process knowledge for dealing with a particular type of change.
Goal: Create reusable change knowledge Task: Abstract from change attributes The analysis of many similar change exemplars (similar in terms of being instances of the same change use-case) allows us to define general or typical values for change attributes. The change attributes we have initially focussed on are those in the severity category i.e. time and cost of change (e.g. a data-change will typically take between 15-20 man-hours to complete). It should be noted that productivity models are based on historical data [21]. We are essentially doing a similar thing, but at an earlier stage in the metrics collection process. To give an example, customers often ask for changes to a CCS after it has been installed. One problem for managers at SDS is estimating what the time and cost of the change will be. Presently, SDS relies on management expertise to carry out estimation. However, we can now facilitate this estimation process by providing typical time and cost figures based on past change exemplars. We expect the reliability of these figures to improve over time as the sample size for the number of changes increases.
Goal: Study the impact of change to software documentation Task: Extend traceability in change scenarios The change scenarios described earlier help to elicit the change process carried out by staff at SDS. We have found that change scenarios are also a good way of validating change processes against the documentation architecture. For example, when a developer at SDS says 'update requirements document' in a change scenario, we are able to question what part of the requirements document, i.e. the documentation unit, and then use documentation dependencies to ascertain what checks need to be done on related parts of the documentation. If we had not done this, changes addressed and documented 'locally' but not in other dependent parts of the document architecture may leave the documentation in an inconsistent state. From this we are able to formulate a number of change heuristics of the form:
231
WHEN change IS A screen-change THEN UPDATE screen-requirements IN CCS-requirements-document AND REQUIREMENTS-CHECK user-interface-requirements IN CCSrequirements-document AND REQUIREMENTS-CHECK high-level-functional-requirements IN CCSrequirements-document AND DESIGN-CHECK screen-layouts IN CCS-design-document... Change heuristics formalise the 'what' to change, 'what' to check and 'when' to check. We see change heuristics as essential in system maintenance situations where the original CCS developer is not the same person who carries out the maintenance. We also believe that change heuristics (though not necessarily of the same form as here) are central to providing knowledge-based support for the change process.
6.4 Sector 4: Change management Goal: Manage change on a specific project Task: Tailor the generic change management process Change management is the fourth sector of the change cycle and is concerned with the management of change on a specific project. The general management process is outlined in Figure 6, which begins with a change request from the customer.
Figure 6 General change management process At the centre of the change management process is the body of change knowledge which is at the centre of the change cycle. We propose that projects tailor the change management process for specific projects in order to define a standard change man-
232
agement process. Such processes can be used to provide practical guidance for dealing with change, especially when encoded within a process-driven softwareengineering environment.
Goal: Resolve viewpoint problems of change Task: Issue and use a 'Change processes' document One of the problems that we have come across is that change is viewed differently from different viewpoints. For example, the 'salesperson' that interacts with the customer might consider a change to be relatively 'easy'. However, the same change might be considered extremely difficult from a developer viewpoint, e.g. because the developer has made some (not unreasonable) assumption. In this kind of case, there is a mismatch between the individual perceptions of change. Resolving viewpoint problems requires working towards a common understanding of change processes between members of a project. One approach is to produce and use a 'change processes' document detailing the kind of change knowledge that we have described in this paper (change use-cases, change attributes etc.). Such a document could be used as a reference document in different scenarios, e.g. change estimation and customer negotiation.
7. Tool Support for Change Captured change knowledge can be exploited in automated or semi-automated tool support for change. We are currently investigating the architectural requirements for tool support for change in SDS. A proposed 'high-level' architecture for a change environment, based on our work with the change cycle at SDS, is shown in Figure 7. There are two types of users, one type who is concerned with the input of change knowledge (post project perhaps), and the other type who is looking for help during a current project.
Figure 7 Proposed high-level architecture for a change environment
233
The change-environment is comprised of three kinds of tools. First, change knowledge input tools which allow the input of change data and knowledge, such as a network editor for 'drawing' out a document architecture. Second, a change knowledge base (KB) for storing change knowledge (though we have not yet worked out the exact representation for the KB). Third, change guidance tools that are able to process change knowledge and provide guidance on a specific project. We see three main change-guidance tools: a change matcher, process guider and estimator. The change matcher matches helps the user match the current change to a change use-case in the change KB. The process guider assists the user by proposing steps for dealing with the change. These process steps are derived from the event trace in the change usecase and by tracing documentation dependencies in the document architecture. The estimator is a tool that is able to generalise from a large collection of change instances. For example, one function of the estimator would be to produce the average time it takes to complete a particular kind of change.
8. Evaluation Approach Our evaluation approach is loosely based around a Goal-Question-Metric [2] paradigm, and on two kinds of cycle which 'track' the process cycle (Figure 8).
Figure 8 Improvement and validation cycles The two kinds of cycle are:
1.
2.
Improvement cycle. This is concerned with the establishment of improvement goals. Typical goals that we have tbund relevant include (GI) reduction in the overall level of change, (G2) reduction in the level of change for a particular type of change, (G3) faster turnaround in completing change requests from the customer. Validation cycle. This is concerned with the establishment of metrics or indicators (because it might be difficult to metricate some aspect of an improvement goal) that supports the improvement goals. Metrics and indicators include (M1) total number of changes (supports G I), (M2) total number of changes pertaining
234
to a particular change use-case category (supports G2) and (M3) average effort taken to complete a change (supports G3). At the time of writing, we have just initiated the collection of metric data. However, we have found that establishing appropriate metrics for change is in itself a nontrivial problem for which there is little discussion of in the current literature. The identification of appropriate and meaningful (to SDS) metrics is part of our on-going work. Our initial experiences suggest that insightful metrics on change is unlikely to be supported without established change procedures in place within the organisation, as exhibited by detailed change request forms and change monitoring tools (as in the change environment proposed earlier). Essentially, we can not do any detailed reasoning on change if the change data is not there in an amenable form. Part of our strategy here has been to choose change attributes that reflect the kind of analysis that we wish to perform.
9. Conclusions This paper has described efforts at change analysis and management in a commercial setting. The starting point for our work was the fact that the systems being developed all pertained to the same domain (namely, the CCS domain). This has encouraged us to take a domain-specific approach to change analysis and management, which differs in emphasis from the existing work on change we reviewed in Section 3. Our main contribution to this area is the 'change cycle', which attempts to provide a concise and integrated view of the change process. We have shown how the change cycle has been applied to address aspects of the change problems in the CCS domain at SDS. One area that we feel we need to elaborate more on in our work is the relationship between the change process and perceived models of the software process, such as Waterfall or Prototyping. This is part of our on-going work to develop a 'changeenhanced' version of a prototying-centred software development approach. As we noted in Sections 7 and 8, tool support for the change process and change metrics are two further areas of on-going work which we hope to report on in more detail in future papers.
10. References 1.
.
.
Avellis, G. (1992), CASE support for software evolution: A dependency approach to control the change process. In proceedings of 5 th International Workshop on Computer-aided software engineering, pp.62-73, Montreal, Canada, July 1992. Basili, V. and Weiss, D. (1984), A method for collecting valid software engineering data, IEEE Transactions on Software Engineering, November 1984, pp.728-735. Bennett, K. (1996), Software evolution: past, present and future, Information and software technology, 38:673-680, 1996.
235
4. 5. 6. 7. 8.
9.
10. 11. 12. 13.
14.
15. 16. 17. 18.
19. 20. 21. 22.
Chung, L., Nixon, B. and Yu, E. (1997), Dealing with change: an approach using non-functional requirements, Journal of Requirements Engineering, 1(4), 1997. Curtis, B., Kellner, M.I. and Over, J. (1992), Process modelling, Communications of the ACM, 35(9), 1992. Fickas, S. and Nagarajan, P. (1988), Critiquing software specifications, IEEE Software, November, 1988. Gilb, T. (1988), Principles of Software Engineering Management, AddisonWesley, England. Han, J. (1997), Supporting impact analysis and change propagation in software engineering environments, In Proceedings of the 8th IEEE International Workshop on Software Technology and Engineering Practice, London, UK, 14-18 July, 1997. Harker, S., Eason, K. and Dobson, J. (1993), The change and evolution of requirements as a challenge to the practice of software engineering, In proceedings of the IEEE international symposium on requirements engineering (RE'93), San Diego, California, 1993. Jacobson, I., Griss, M and Jonsson, P. (1997), Software reuse: architecture, process and organisation for business success, ACM Press, New York. Kaiser, G., Feiler, P. and Popovic, S. (1988), Intelligent assistance for software development and maintenance, IEEE Software, 5(3):40-49, May 1988. Lam, W. (1997), Achieving Requirements Reuse: a Domain-Specific Approach from Avionics, Journal of Systems and Software. 38(3): 197-209, 1997 Lam, W., McDermid, J.A. and Vickers, A.J. (1997), Ten Steps Towards Systematic Requirements Reuse (expanded version), Journal of Requirements Engineering, 2:102-113, 1997. Lam (1998a), Managing requirements evolution and change, IFIP WG2.4 Working conference: Systems implementation 2000: languages, methods and tools, Berlin, Germany, February 23-26, 1998. Lam, W. (1998b), A Case-study of Requirements Reuse through Product Families, Annals of Software Engineering (to appear). Lehman, M. (1996), Feedback in the software evolution process, Information and Software technology, 38:681-686, 1996. Madhavji, N. (1991), The Process Cycle, Software Engineering Journal, September, 1991. Madhavji, N. (1997), Panel session: impact of environmental evolution on requirements changes, In Proceedings of the 3rd IEEE International Conference on Requirements Engineering, 1997. Podgurski, A. and Clarke, L. (1990), A formal model of program dependencies and its implication for software testing, debugging and maintenance, IEEE Transactions on Software Engineering, SE-10(4):352-357, 1984. Pohl, K., Domges, R. and Jarke, M. (1997), Towards method-driven trace capture, 9 th International Conference, CaiSE'97, Barcelona, Spain, June 1997. Putman, L. and Myers, W. (1992), Measures for excellence, Reliable software on time, within budget, Yourdon Press, Prentice-Hall, New Jersey, 1992. Sugden, R. and Strens, M. (1996), Strategies, tactics and methods for handling change, In Proceedings of International IEEE Symposium and Workshop on Engineering of Computer-Based Systems (ECBS '96), Friedrichshafen, Germany, March 11-15.
236
23. Yu, E. (1997), Towards modelling and reasoning support for early-phase requirements engineering, In Proceedings of the 3rd IEEE International Conference on Requirements Engineering, 1997. 24. Weiser, M. (1984), Program slicing, IEEE Transactions on Software Engineering, SE-10(4):352-357, 1984. 25. Zave, P. (1986), Let's put more emphasis on prescriptive methods, ACM Sigsoft Software Engineering Notes, 11(4):98-100.
A F i l t e r - M e c h a n i s m for M e t h o d - D r i v e n Trace Capture Ralf D6mges, Klaus Pohl, Klaus Schreck RWTH Aachen, Lehrstuhl lnformatik V, 52056 Aachen, Germany email: {doemgeslPohllsehreck}@i5.informatik.rwth-aachen.de Abstract: Traceability is a prerequisite for developing high quality (software) systems. Recording and maintaining all available information is too labor intensive and thus by far too expensive. A project-specific definition of the trace information to be recorded and the method fragments (so called trace fragments) to be executed for recording the information provides a solution for this problem. But the amount of traces to be recorded does not only vary from project to project. It also varies between project phases and even within a project phase. As a consequence project-specific trace fragments need to be adapted according to the actual project phase. In this paper we propose a model-based filter mechanism to significantly reduce the required effort to adapt trace fragments. By defining appropriate filters the project manager is able to (dynamically) adapt the project-specific trace fragments to the actual needs. We present an example to highlight the benefits of the approach and discuss possible extensions.
1 Introduction 1.1 Traceability: Definition and Motivation Traceability, more precisely requirements pre- and post-traceability, is recognized by many research contributions and experience reports as an important prerequisite for developing high quality systems (e.g., Collery, 1988; lEE, 1991; Ramesh et al., 1996). Among others, traceability facilitates the development, validation and maintenance of a system, provides invaluable support for the integration of changes, reduces the amount of errors during system development, allows experience-based process improvement, and is important to prove the fulfilhnent of contracts Jarke et al., 1994; Pohl, 1996b. Traceability is thus required by various system development standards (e.g., V-Modell Br~hl and Drtischel, 1993, DoD-2167A DoD-2167A, 1988) as well as quality and process improvement frameworks, like ISO 9000-3 ISO, 1991 and CMM Paulk et al., 1993. Various contributions (e.g., Kaindl, 1993; Gotel, 1996; Yu and Mylopoulos, 1994; Conklin and Begeman, 1988) and traceability fi'ameworks (e.g., Pohl, 1996b; Ramesh et al., 1996) define a large set of information to be captured for enabling traceable system development. Obviously, an approach recording all this trace information in each project and for all system components is by far too time consuming, thus too expensive, and therefore likely to fail Tilbury, 1989.
1.2 Project-specific Trace Capture: Method-Driven Approach To reduce the effort for capturing and maintaining traceability information the traces, i.e., the information which is actually being stored, has to be recorded according to project-specific needs Pohl and D6mges, 1997. Existing prototypical trace environments (e.g., TOOR Pinheiro and Goguen, 1996, PRO-ART 1.0 Pohl, 1996b) as well as existing commercial requirements and sys-
238
tern engineering environments like DOORS Quality Systems & Software, 1996, RDD-100 Ascent Logic Corporation, 1994, RTM Marconi Systems Technology, 1996, or SLATE TD Technologies, Inc., 1996 support the persistent recording and management of trace information. They typically provide a set of generic information types and operations which can be specialized according to project-specific needs. Thus, they empower the project manager to define project-specific trace information, e.g., design decisions and their relations. The environments provide comprehensive consistency checking capabilities and support the retrieval and display of the recorded traces by advanced ad-hoc and pre-defined querying, browsing, and reporting mechanisms. But they do not provide systematic support for recording the defined trace information during the system development process, e.g., the project manager cannot define when (in which situation) a decision should be recorded. Our method--driven approach to trace capture (see Pohl et al., 1997 for details) overcomes this sllortcoming. It supports tile explicit definition of project-specific trace information together with trace fragments which define strategies for capturing the information. Moreover the user is guided in recording the information according to the project-specific trace fragment definitions. This is technically achieved by interpreting the defined trace capture strategies in a process-integrated environment. Based on the interpretation the environment reminds the user about the traces to be captured, enforces (e.g., in project critical procedures), and even automates (whenever possible) the recording of traces I. We use the NATURE process meta model Rolland and Grosz, 1994; Pohl, 1996b to define the project specific trace fragments. The model distinguishes between: 9 Atomic fragments for representing the part of a method definition which can be automated. Atomic fragments have no (externally) visible structure and are "hard-coded" in the tool or environment. By executing atomic fragments traces are (automatically) created and recorded in the trace-repository. 9 Strategy selection fragments for representing the part of the method definition in which the user has to make a decision between at least two alternatives, i.e., alternative trace capture strategies. A trace strategy defines the way how traces are (interactively) created. 9 Composed method fragments (resp., composed trace fragments) for defining complex trace strategies, i.e., for defining a certain order on a set of trace fragments. The project manager uses the three types of fragments to define the project-specific trace strategies. Thereby s/he (implicitly) defines the trace information to be recorded. To support the definition of the trace strategies we distinguish between four information types. Depending on the information type the recording is performed manually and/or automatically. Table 1 depicts the information types and the way an information is typically being recorded. We have integrated the project-specific trace capture strategy into our TECHMOD D6mges et al., 1996 and PRO-ART 2.0 Pohl, 1996a environments. We used both environments in small case studies. The studies showed that reminding the stakeholders about the recording of the project specific trace information resulted in traces of higher quality compared to traces produced by following "paper-based" trace capture guidelines. Moreover, recording of unnecessary trace information was avoided and the work load of the users was (significantly) reduced. I The detailed mechanism required to enable such a project-specific guidance of trace capture based on the interpretation of method fragments is described in Polfl and Weidenhaupt, 1997.
239
information trace type fragment product hzformation(e.g., ER-model) process fragment supplementary product information (e.g., supplementaryprocess design decisions) fragme,lt process observation b~)rmation (e.g., processobservation methodfragment) fragment dependency information (e.gl, interrelate dependencyfragment ER-model and designdecisions)
recording mainly interactive I automated X X
X X
X
Tab. 1. lYace information and corresponding trace fragments 1.3 Project-specific Adaptation of Trace Fragments However, it turned out that tile trace fragments had to be adapted due to two main reasons: 1. Traces vary between project phases. For example, while in the specification phase recording of structured design decisions was demanded, the maintenance phase focused on recording and interrelating change requests and change approval forms. 2. Traces depend on the components developed, e.g., when safety critical and/or very complex components are developed design decisions, meeting notes, conceptual drawings, as well as scenarios must be captured and interrelated. An obvious solution for the adaptation of trace fragments to project-phase-specific needs was to specialize the (existing) trace fragments accordingly. This solution had two significant shortcomings 1. modeling and~or re-modeling of the required trace fragments was a difficult and time consuming task; 2. programming and~or re-programming of atomic fragments was often required. As a consequence the amount of trace fragments maintained in our method base rapidly increased. The trace fragments were almost identical; they differed only slightly in the trace-capture dependent parts. Thus managing the trace fragments got very complicated and maintahfing them consistently was almost impossible.
1.4 Approach and Structure of Paper In this paper we propose afilter mechanism to overcome the above shortcomings. In section 2 we elaborate the main requirements for a filter mechanisnl to significantly reduce the required effort to adapt trace fragments. Providing a filter mechanism is essential to empower an easy and flexible adaptation of fragments to continuously evolving information needs. The filter mechanism described in sections 3 and 4 allows a dynamical adaptation of trace fragments. Filters can be defined for a specific project phase as well as for particular trace fragments. A prototypical implementation of the filter mechanism was integrated into our process integrated environments and applied to small examples (section 5). Finally, we provide a conclusion of the achieved results and give an outlook on future work (section 6). 2 Requirements for a Model-Based Filter Mechanism
2.1 Prevent Storage of Trace Information The filter mechanism has to ensure that certain trace information is not recorded in
240
the trace repository. To avoid the re-programming of atomic fragments the filter mechanism should be able to partially block the output of an atomic fragment from being stored in the trace repository. For example, take the automated recording of process observation information by an atomic fragment. This fragment records all executed trace fragments together with the agent who executed them. Assume that due to contractual or organizational regulations the agents should not be recorded. A trace filter which could block the information about the agent of being stored in the trace repository avoids the re-programming of the atomic fragment.
2.2 Restrict Selection of Trace Strategies The filter mechanism must provide means to restrict the alternatives provided by a strategy selection fragment. Whenever an alternative of a strategy selection Ii"agment should not be used, it should not be offered to the user. For example, a strategy selection li"agment provides four alternatives to the user to justify the creation of a new product version: (1) recording a structured design decision, (2) stating the reasons for the change as informal text, (3) selecting an appropriate part of the project--contract, or (4) indicating the person creating the new version. If during the early project-phases justifications should only be given by informal text or by naming the responsible person the two other alternatives must be removed. 2.3 Prevent Execution of Trace Fragments The filter mechanism must allow to prevent the execution of a trace fragment. Whenever the entire output information of a trace fragment should not be recorded the execution of the fragment has to be prevented; especially if a trace fragments requires user interactions. For example, if design decisions should not be recorded during early project phases the execution of the trace fragment for recording the decision has to be prevented. 2.4 Enable Filter Definitions for the Overall Projects and Project Phases In addition to filter definitions for a single trace fragment the filter mechanism should provide means to define filters which affect all trace fragments executed in a project phase. For example, to capture no process observation information during the proposal phase whereas their recording is mandatory during the requirements engineering and design phases it should be possible to define a filter for an entire project phase. 2.5 Empower Nested Filter Delinitions Trace fragments can be nested forming composed trace or strategy selection fragments. This requires propagation rules for filters defined for nested trace fragments. For example, assume that capturing process observation information is explicitly prohibited for a composed trace fragment. Consequently, none of the trace fragments used for defining the composed trace fragment should record process observation information, even if defined differently by filters of these trace fragments. 2.6 Enable Enforcement of Information Storage and Fragment Execution Within a nested trace fragment various information types can be blocked, alternative strategies can be restricted, and/or the execution of trace fragments can be prevented by defining the appropriate filters. For a composed trace or strategy selection fragment the filter mechanism must provide means to ensure the recording of a certain type of
241
information, the offering of particular alternative trace strategies, and/or the execution of specific trace fragments regardless of the filters defined for the fragments which are (constituent) parts of the composed fragments. For example, assume that the recording of automated dependency information is blocked by a filter F defined for a trace fragment tf. To assure the recording of this information in a composed trace fragment the project manager should be able to define a filter F' which "replaces" F; i.e., which assures the recording although the nested filter F prohibits the recording.
3 Information, Strategy, and Method Filters To fulfill the requirements defined above three filter types are required: information flters which prevent already produced information from being stored in the repository (section 3.1), strategy filters which restrict the available trace capture st,'ategies delined by a strategy selection fragment (section 3.2), and method filters which prevent trace fragments from being executed (section 3.3).
L I'y
I"
I trgme.t II ,,gmen,
fragment
Fig. 1. Filters types
These three filter types are defined as specialization of the conccptfilter (see figure 1). The scope defines if a filter is valid within a project phase or a trace fragment (see requirement 2.4). A filter can be related to more than one scope. If a filter should be applied in particular project phase(s), the filter is associated to the phase(s) by using the has scope association. If a filter is associated to all project phases, the filter is applied to the overall project. If a filter should only be valid for a particular trace fragment, the filter is related to the fragment using the has scope association. 3.1 Information Filters h~formation filters prevent information types from being stored persistently although the corresponding trace fragments are executed and their output information is produced (figure 2; requirement 2.1). The type of information to be blocked by a given information filter is defined using the association block information type defined between the class information type and the class information filter. The project phases and trace fragments to which an information filter should be applied are related to the information filter using the has scope association. An information filter can be used to avoid the recording of a particular information produced by an atomic fragment; but it can also be used to avoid the recording of a particular information during a project phase. An information filter can block more than one information type.
242
Fig. 2. Information filters Information filters are essential for influencing tile itdbrmation capture of atomic fragments. Inlbrluation filters are very well suited for blocking process observation information which is typically recorded automatically. They are generally applicable for dependency information, especially if dependencies are created automatically. Information filters should not be used to filter supplementary product information, since these information types are usually created interactively. 3.2 Strategy Filters Strategy filters restrict the available alternatives of strategy selection fragments (figure 3; see requirement 2.2). By applying strategy filters to a strategy selection fragment only a subset of the defined alternatives is offered to the user. The alternative(s) which should not be offered are defined using the association block alternative defined between the class strategy filter and the association class alternative defined between the class strategy selection fragment and the class trace fragment (figure 1). A strategy filter can be related to more than one alternative fragment.
Fig. 3. Strategy filters The scope of the filter is defined by associating the appropriate project phases and/or trace fragments with the filter. If a strategy filter is associated to one or more project phases, the blocked alternatives are not offered within the defined phases; but are offered within other phases. If it is associated to a trace fragment, the blocked alternatives are not offered whenever the strategy selection fragment is executed within the trace fragment. 3.3 Method Filters Method filters prevent the execution of one (or more) atomic fragment, strategy selection fragment, or composed trace fragment (figure 4; see requirement 2.3). The trace fragments to be blocked are defined using the association block fragment defined
243
between the class metlmdfilter and the class trace fragment (figure 1). A method filter can block more than one method fragment. The scope of the filter is defined by using the/ms scope association to relate project phases and/or trace fragments with the filter. If a method filter is associated to one or more project phases, the blocked fragment is never executed within the defined phases; but it will be executed in other phases. If it is associated to a trace fragment, the blocked fragment is not executed within the associated fragment.
Fig. 4. Method filters
Method filters should be used to adapt the interactive capture of traces, i.e., the recording of supplementary products and interactively created dependency information. They provide no means to change the internal control-flow of composed trace fragments. This can (only) be achieved by a manual adaptation of the fragment definition. 4 Nesting Information,
Strategy, and Method Filters
The trace fragments which define project-specific trace capture can be nested by defining composed trace or strategy selection fragments. Each of these nested trace fragments tfcan be defined as the scope for a set of filters (denoted asfilterset(tJ)). As a consequence, the filters defined for the trace fragments are also nested. Consequently, we need to define propagation rules to determine the filters which have to be applied if a particular trace fragment is executed (section 4.1).
A
/\
~lom~ton/g~ I
~
aX~,nn~
strategy selection
I
a
Fig. 5. Filter modes
The (propagated) filter can not be used to enforce the recording of information types, that alternatives of a strategy selection fragment are offered, or the execution of particular trace fragments. For example, assume a filter associated to a trace fragment which is contained within a composed trace fragment. The filter defines to block
244
the recording of a particular information type. If we want to ensure that the trace information will be recorded whenever the nested fragment is executed, we must provide means to define which information should be recorded during the execution of a composed trace fragment regardless of the filter definitions associated to its contained fragments. We therefore introduce afilter mode attribute (figure 5). Using this attribute the project maalager is able to define explicitly that a filter prevents or permits (a) the defined information type to be recorded (information filter); (b) a certain alternative to be offered (strategy filter); (c) a trace fragment to be executed (method filter). Due to the filter mode the filters which have to be applied to a trace fragment may contradict each other. In section 4.2 we define these contradictions and describe how they can be resolved. Finally, we provide some rules towards a systematic application of trace filters (section 4.3). 4.1 Fragment Scopes and Valid Filter Sets
For nested filters we determine the valid filter set of a trace fragment. The valid filter set consists of the filters to be applied to a fragment whenever the fragment is executed. We first define a containment-relationship between trace fragments: 9 A trace fragment tf directly contains another trace fragment tf' (denoted as tf' E tf) iff a.) b.)
tf is a composed method fragment and there exists a composed of link between tf and tf'. tf is a strategy selection fragment and there exists an alternative link between tf and tf'.
In other words, tf' is used to define tf. 9 A trace fragment tf indirectly contains another trace fragment tf' (denoted as tf' E tf) iff tf' E tf or 3tf" with tf" E tf A tf' E tf" Figure 6 depicts examples of this containment relationship for some trace fragments, e.g., tf2Etfs, tflEtf2, and tfl Etfs.
Fig. 6. Containment and scopes of trace fragments To determine a valid filter set we need to define the scope of a nested trace fragment tf. The scope of tfis a set of trace fragments which directly or indirectly contain each other. The scope of tf is defined relative to a particular trace fragment tf' because tf may be (directly or indirectly) contained in more than one trace fragment. For example, a trace fragment to record design decisions can be used within a composed trace fragment which defines how to integrate changes into an existing specification as well as within a trace fragment which describes the creation of a new version of a specification. In figure 6 are if1 E tf2 and tflE tfs. Consequently, a trace fragment tf usually has more than one scope.
245
We define
scope(tf, t f ' ) = (tfko . . . . , t f k , ) with t f = tfko, tfk,_, ~ tf~,,Vi ~ { l , . . . , n } , and t f k , = tf'. If t f = t.f~o = t A , = t.f' then seope(t.f, t.f') = (t f ) = (t.fko) = (t.fk,) = ( t f ' ) . Figure 6 depicts some examples: scope(tft,tfs) = (tft,tf2,tfs) and scope(tft,tfs) = (tft,tfs) Based on filterset(tJ) we define the valid filter set of a nested trace fragment tf (denoted as VFS(tf, tf')) as the union of the filtersets of its corresponding scope: VFS(tf, tf') = filterset(scope(tf, tf') ) andfilterset((tfk~ . . . . , t A , ) ) = U filterset(lfk,). i--~01...111
VFS(tf, tf') contains also tile filters which are associated with the project phase ill which tf (and the fragments of scope(tf, tf')) is actually executed. By defining and using the scope of a trace fragment to deline its valid filter set we have established the propagation rules to determine the filters which have to be applied when a particular trace fragment is executed. 4.2 Contradicting Information, Strategy, and Method Filter Definitions A valid filter set may contain filters having contradicting filter modes. For example, if the recording of design decisions is enforced on the level of a composed trace fragment (i.e., the filter mode is set to permit) but prohibited on the level of trace fragments which are contained in the composed trace fragment it is not clear which filter should be applied. We thus need to define those contradictions and how to resolve them. In the following we denote filterobjects(F) of an information, strategy, or method filter F as the set of all information types, alternatives, or trace fragments to be permitted or prevented by F. Contradictions between two filters of the same type are defined and resolved as follows:
9 Let IF and I F ' b e two information filters, filterobjects(IF) = {itt ..... itn}, n > !, and filterobjects(IF') = {it'j ..... it',. }, m > 1. The filter definitions of IF and IF' are contradictory iff the filter mode of IF is permit, the filter mode of IF' is prevent (or vice versa), and {ih ..... it,,} N {it'j ..... it'm} = { itkt , . . .,itk~}, l > 1. If IF, IF' E VFS(tf, tf') are two contradictory information filters, scope(tf, t.f) = {tfl ..... tf,.}, IF' E filterset(t.~), IF E filterset(~) and i > j, the filter definitions of IF' for { i t k , , . . . , i t k z } replace the filter definitions of IF for { i l k ~ , . . . , itk~ } by removing them from VFS(tf, tf') (figure 7 a.)).
9 Let
F
and
F'
Fig. 7 Contradictory filter definitions be two strategy filters (or two
method
filters),
246
filterobjects(F) = {tff l ..... t f n}, n > 1, and filterobjects(F') = {tf't ..... tfl:'m}, m > 1. The filter definitions of F and F' are contradictory iff the filter mode of F is permit, the,, filter mode of F' /,,is prevent (or vice versa), and , /~, > { t f , ..... t f n } n {tff , ..... t f ,,} = { t f k , , . . . , t f k , } , l _ I. If F, F' E VFS(tf, tf') are two contradictory strategy filters (or method filters), scope(tf, tf') = {tft ..... ~ } , F' E filterset(tfi), F E filterset(tfj) and i > j, the filter definitions of F' for {tf~'l,..., t f~i} replace the filter definitions of F for { t f kf,' , . . . ,tf~i} by removing them from VFS(tf, tf') (figure 7 b.)). If t~ = t~ (i.e., i = j) the same fragment is associated with contradicting filter definitions. In this case the project manager has to decide which of the filter definitions should be replaced. Two filters of differem types are contradictory il'f IF is an infornlation filter, F is a strategy filter (or method filter), filterobjects(IF) = {itt ..... it,}, filterobjects(F) = {tf't ..... tf',,,.}, the filter mode of IF is permit, the filter mode of F is prevent, and there exists an atomic fragment af E tf't, l E {1 ..... m'} which produces { i t j , , . . . , itjk } C { i t 1 , . . . , it,}. If IF, F E VFS(tf, tf') are two contradicting filters, where IF is an information filter and F is a strategy filter (or method filter), scope(tf, tf') = {tfl ..... if,, }, IF E filterset(tfi), F E filterset(tf)) and i > j the contradiction can not be resolved automatically but the project manager has to decide how to resolve the contradiction. Thereto s/he needs to determine the trace fragment tf't which contains the affected atomic fragment af E tf't and has to figure out how to adapt the filter definitions of IF and F, e.g., by preventing the execution of all trace fragments which are contained within tf't except af. There will be no contradictions between the filter definitions of a method filter M F and a strategy filter SF. We demand that alternatives of strategy selection fragments can only be prevented by strategy filters (see section 4.3) and strategy filters are only able to restrict the alternatives of a strategy selection fragment. Even if a method filter defines to prevent the execution of an alternative it is still offered to the user. If the filter definitions associated with the project phase in which a trace fragment tf (and the fragments of scope(tf, tf')) is actually executed contradict with VFS(tf, tf') the filter modes of the project phase generally replace the filter modes of the trace fragments. The above definitions can be used to analyze the filters defined for the trace fragments. Contradicting filters can thus be detected before the trace fragments are actually applied. The project manager can resolve the contradictions before the fragments are executed during a project. 4.3 Rules for applying Filters Based on our experience we provide some rules for applying information, strategy, and method filters:
Apply filters not to product information: Product information should never be affected by filters. Product information is the main output of the development process. Hence, it makes no sense to block their recording. For example blocking product information during the development of a Entity-Relationship model would lead to an incomplete and inconsistent model. Filters should only affect the recording of supplementary product, process observation, and dependency information. If a change in product information is required (e.g., define inheritance (links) in En-
247
tity-Relationship--diagrams) new fragments have to be introduced and/or existing fragments have to be adapted. This definition and/or re--definition of a method is not within the scope of a filter mechanism.
Apply information filters only to automated trace fragments: If the information of interactive trace fragments is blocked by information filters it is very likely that users reject to enter the information next time. This might lead to the rejection of the entire filter-based approach for capturing traces. Information filters should thus never be used to block interactively entered information. Apply method or strategy filters when complete output information is blocked: A fragment whose complete output is blocked by (nested) information filters should not be executed. Instead, a method filter should be defined to prevent the execution, or if the fragment is an alter,mtive of a strategy selection fragment, all appropriate strategy filter should be defined. Apply method filters when all alternatives of a strategy selection fragment art, prevented: If the entire set of alternatives of a strategy selection fragment is prevented by (nested) strategy filters, the fragment should not be executed. Instead of defining strategy filters which block all alternatives, a method filter should be defined to prevent the execution of the strategy selection fragment. Check effects on composed trace fragments: If any kind of filters prevent the storage of information or the execution of a trace fragment within a composed trace fragment, the project manager must check if the blocking of the information (or the fragment) does not lead to a "deadlock". In other words, s/he must assure that a composed trace fragment could be executed although a trace fragment is blocked and/or information is not recorded. In the case of a detected deadlock s/he must change the control flow of the composed trace fragment. Do not apply method filters to block alternatives of strategy selection fragments: Method filters should not be misused as strategy filters, i.e., they should not be used to block an alternative of a strategy selection fragment. By defining a strategy filter, the alternative is not offered to the user, whereas in the case of a method filter, the alternative is offered to the user. The user can choose the alternative, but the chosen alternative will not be executed. Together with the scope and contradictions defined in sections 4.1 and 4.2 the rules provide the basis for developing an environment which supports the project manager in defining consistent trace filters of any type. 5 Model-Based Filtering: An Example We illustrate our model-based filter mechanism using a small example. The composed trace fragment integrate change request guides the application engineer during the integration of changes (figure 8). The application engineer is first reminded to justify the changes. The strategy selection fragment select justification object defines three alternative strategies for the justification: (1) to select appropriate parts of a contract; (2) select the stakeholder who initiated the change; or (3) to select a specific design decision. A process observation fragment automatically records the execution of the strategy selection fragment and the chosen alternative. During the integration of changes an automated dependency step relates the object representing the justification with the modified and/or created specification parts.
Fig. 8. Composed trace fragment bitegrate change requests (simplified).
The fragment described above is reused for the proposal phase of attother project. The project management decides that in this project it is suflicient to justify the change by stating the responsible stakeholder. In other words, the two other alternatives of the strategy selection fragment should not be offered. Since two of the three alternatives of the strategy selection fragments are blocked, the chosen alternative needs not to
be recorded by the process observation step. Moreover the project manager decides that no dependencies should be created between the stakeholder initiated the changes and the modified or created specification parts. We use our filter mechanism to adapt the method fragment integrate change request according to the new requirements of the project manager. We define 9 one strategy filter which blocks the alternatives select contract parts and select design decisions of the strategy selection fragment select justification object; 9 one method filter which prevents the execution of the atomic fragment create dependency; instead of associating an information filter with the atomic fragment to block its entire output; 9 one information filter which blocks the recording of the information about the chosen alternatives. This filter is associated to the record strategy selection
Fig. 9. Adapted trace fragment integrate justified change (simplified) The application of these filters leads to the trace fragment(s) depicted in figure 9. The parts of the trace fragment which are not executed, i.e., prevented by the filters, are depicted in grey. This changes could be achieved without any re-modeling of the composed trace and strategy selection fragments and without any reprogramming of the atomic method fragments.
249
6 Conclusion and Future W o r k Our approach to method-driven trace capture Pohl et al., 1997 enables the definition of project-specific trace information and trace capture strategies. Based on this definitions the user is guided in capturing the required project-specific trace information. Originating from its application in case studies two main shortcomings of the approach were recognized: adapting trace fragments to varying traces during a project required a significant effort for (re-)modeling and (re-)programming; managing and maintaining the trace fragments became ahnost impossible due to redundant parts of the introduced fragments and a rapidly increasing amount of trace fragments. The filter-mechanism presented in this paper avoids the two shortcomings. Based on a set of requirements for trace filters we have defined three types of filters: 9 information filters block certain information types fl'om being stored in the
repository; 9 strategy filters restrict the alternative trace strategies offered to the user; 9 method filters prevent a trace fragment from being executed.
A filter can be defined for particular project phases or specific trace fragments. The filter definitions influence the recording of the traces during a project phase and/or during the execution of a trace fragment. To enforce the recording of certain information we have defined two filter modes: prevent and permit. We defined propagation rules for nested filters to determine all filters to be applied for a trace fragment whenever it is executed and specified how to resolve resulting contradictory filter definitions. To support the systematic definition of filters we provided a set of rules for their application. The filter mechanism was validated by integrating it into the TECHMOD and PRO-ART 2.0 environments and by applying it to small examples. Early experience confirms that trace filters significantly reduce the necessary effort to adapt trace fragments and facilitates the management and maintenance of the method base. The development of tool support for the definition and application of filters will be focus of our future work. Such support should employ the defined rules for applying filters and provide mechanisms to check the effects of filters on the trace fragment definitions. Acknowledgments This work was supported by the DFG-Projekt "Prozel3-1ntegratio,1 yon ModellierungsArbeitspltitzen", the ESPRIT Long Term Research Project 21903 CREWS (Cooperative Requirements Engineering With Scenarios), and the DAAD/ACLS/NSF program "Technische und empirische Grundlagen der Requirements Traceability: Modelle und Mechanismen". The authors are grateful to their colleagues P. Haumer, M. Jarke, K. Weidenhaupt, and S. Zlatintsis for many fruitful discussions and contributions.
References Ascent Logic Corporation, 1994 Ascent Logic Corporation. RDD-100Marketing Brochu~, 1994. Br6hl and Dr6schel, 1993 A.P. Br6hl and W. Dr6schel. Das V-Modell. OldenbourgVerlag, 1993.
250
Collery, 1988 A. Collery. Traceability, the New Strategic Challenge for Companies, its Tool, Character Reading in Industrial Circles. In Proc. of the 19th Intl. Symposium on Automotive Technology and Automation, with Particular Reference to Cell Control and Quality Management Systems for the Manufacturing Industries, volume 1, pages 251-260, Monte Carlo, Monaco, October 1988. Allied Automation. Conklin and Begeman, 1988 J. Conklin and M.J. Begeman. glBIS: A Hypertext Tool for Exploratory Policy Discussion. ACM Transactions on O/rice Information Systems, 6(4):303331, 1988. DoD-2167A, 1988 DoD-2167A. Military Standard: Defense System Software Development. 1988. U.S. Dept. of Defense. DOmges et al., 1996 R. Dtimges, K. Pohl, M. Jarke, B. Lohmann, and W. Marquardt. PROART/CE m An Environment for Managing Chemical Process Simulation Models. In Proc. of the lOth Europ. Simulation Multiconference, pages 1012-1017, Budapest, Hungary, June 1996. Gotel, 1996 O. Gotel. Contribution Structures .fi~r Requirement.~ Engineering. PhD thesis, Imperial College of Science, Technology, and Medicine, London, England, 1996. lEE, 19911 lEE. Proceedings of tile lEE Colloqtdmn on Tools, 7'echniques for Maintaining Traceability During Design, London, England, December 1991. ISO, 1991 ISO. IS09000-3: Quality Management and Quality Assurance Standards. International Institute for Standardization, Genf, Switzerland, 1991. Jarke et al., 1994 M. Jarke, K. Pohl, C. Rolland, and J.-R. Schmitt. Experience-Based Method Evaluation and Improvement: A Process Modeling Approach. In IFIP WG 8.1 Conference CRIS '94, Maastricht, The Netherlands, 1994. Kaindl, 1993 H. Kaindl. The Missing Link in Requirements Engineering. ACM SIGSOFT Software Engineering Notes, 19(2):30-39, 1993. Marconi Systems Technology, 1996 Marconi Systems Technology. RTM (Requirements & Traceability Management) - Marketing Information, 1996. Paulk et al., 1993 M. Paulk, B. Curtis, M. Chrissis, and C. Weber. Capability Maturity Model for Software: Version I.I. Technical Report SEI-93-TR-24, Software Engineering Institute, Carnegie Mellon University, Pittsburg, Pennsylvenia, USA, February 1993. Pinheiro and Goguen, 1996 F.A.C. Pinheiro and J.A. Goguen. An Object-Oriented Tool for Tracing Requirements. IEEE Software, pages 52--64, March 1996. Pohl and D6mges, 1997 K. Pohl and R. D6mges. An Environment for Model-Based Trace Capture. In Proc. of the Intl. Conf. on Software Engineering and Knowledge Engineering, Madrid, Spain, June 1997. Pohl and Weidenhaupt, 1997 K. Pohl and K. Weidenhaupt. A Contextual Approach for Process-Integrated Tools. In Proc. of the 6th Europ. Software Engineering Conference, Ziirich, Switzerland, September 1997. Pohl et al., 1997 K. Pohl, R. D~Smges,and M. Jarke. Towards Method-Driven Trace Capture. In Proc. of the 9th Intl. Co:~ on Advanced lnfi)rmation Systems Engineering, Barcelona, Spain, June 1997. Pohl, 1996a K. Pohl. PRO-ART: Enabling Requirements Pre-Traceability. In Proc. of the 2nd Intl. Conf. on Requirements Engineering, Colorado-Springs, Colorado, USA, April 1996. Pohl, 1996b K. Pohl. Process Centered Requirements Enghwering. RSP by J. Wiley & Sons Ltd., England, 1996. Quality Systems & Software, 1996 Quality Systems & Software. DOORS (Dynamic Object Oriented Requirements System) - Marketing Information, 1996. Ramesh et aL, 1996 B. Ramesh, C. Stubbs, T. Powers, and M. Edwards. Implementing Requirements Traceability: A Case Study. Annals of Software Engineering, 9:1-19, 1996. RoUand and Grosz, 1994 C. Rolland and 0. Grosz. A General Framework for Describing the Requirements Engineering Process. In Proc. of the Intl. Conf. on Systems, Man, and Cybernetics, San Antonio, Texas, USA, October 1994. IEEE Computer Society Press. TD Technologies, Inc., 1996 TD Technologies, Inc. SLATE (System Level Automation Tool for Engineers) - Marketing Information, 1996. Tilbury, 1989 A.J.M. Tilbury. Enabling software traceability. In Proc. of the lEE Colloquium on The Application of Computer Aided Software Ensineering Tools, pages 7/1-7/4, London, England, February 1989. Yu and Mylopoulos, 1994 E. Yu and J. Mylopoulos. Using Goals, Rules, and Methods to Support Reasoning in Business Process Reengineering. In Proc. of the 27th Hawaii Intl. Conf. on bystem Sciences, volume IV, pages 234-243, Maui, Hawaii, USA, January 1994.
Subject-Based Organization of the Information Space in Multi-database Networks Michael P. Papazoglou I and Steven Milliner 2 1 Tilburg University, INFOLAB, P.O. Box 90153, 5000 LE Tilburg, The Netherlands [email protected] 2 Queensland University of Technology, School of Information Systems, GPO Box 2434, Brisbane QLD 4001, Australia [email protected] A b s t r a c t . Rapid growth in the volume of network-available data, complexity, diversity and terminological fluctuations, at different data sources, render network-accessible information increasingly difficult to achieve. The situation is particularly cumbersome for users of multi-database systems who are expected to have prior detailed knowledge of the definition and uses of the information content in these systems. This paper presents a conceptual organization of the information space across collections of component systems in multi-databases that provides serendipity, exploration and contextualization support so that users can achieve logical connections between concepts they are familiar with and schema terms employed in multi-database systems. Large-scale searching for multi-database schema information is guided by a combination of lexical, structural and semantic aspects of schema terms in order to reveal more meaning both about the contents of a requested information term and about its placement within the distributed information space.
1
Introduction
T h e dramatic growth in global interconnectivity has placed vast amounts of d a t a within easy reach. At the same time it has made on-demand access to widelydistributed data a natural expectation for a variety of users. A limiting factor however, is the difficulty in providing coherent access and correlation of d a t a t h a t originate from diverse widely-distributed d a t a sources. This is an involved process not only due to the sheer volume of information available, but also because of heterogeneity in naming conventions, meanings and modes of d a t a usage. Differences in data descriptions, abstraction levels, and precise meanings of terms being used in disparate d a t a sources do not yield well at all to automation. These problems are compounded by differences in user perceptions and interpretations, and variations t h a t m a y occur at autonomous database sites over time. Users are thus presented with the problem of gaining adequate knowledge of a potentially huge, complex dynamic system, in order to access and combine information in
252
a coherent and logical manner. Yet multi-database systems demand from users
prior detailed knowledgeof the definition and uses of their underlying data 24. This expectation is quite unreasonable in large distributed systems. The focus in multi-database systems is on query processing techniques and not on how to discover where the actual schema elements in the component systems reside. No particular attention is paid to how schema items are structured, what they mean and how they are related to each across component database schemas. The user's perception of the information content in networked databases is that of a vast space of information in a large flat, disorganized set of database servers. In contrast to this, our approach to searches for widely distributed information concentrates on providing a dynamic, incremental and scalable logical organization of component database sources, and search tools that are guided by this organization. We view user interaction with a multi-database space as comprising two major phases, the: s c h e m a i n f o r m a t i o n d i s c o v e r y p h a s e where users systematically explore the multi-database space to locate potentially useful databases, and the d i s t r i b u t e d q u e r y / t r a n s a c t i o n p h a s e where the requested data sets are retrieved from the candidate databases. We consider the development of a methodical, scalable search process critical to the successful delivery of information from networked database systems. Hence, in order to provide users with tools for the logical exploration of distributed information sources a four step process, termed information elicitation is introduced and includes: (i) Determining the information needs of users by means of different term suggestions; (ii) Locatingcandidate database sources that address these needs; (iii) Selecting schema items of interest from these sources; and finally, (iv) Understanding the structure, terminology and patterns of use of these schema items which can subsequently be used for querying/transaction purposes. The very nature of this process suggests that we should provide facilities to landscape the information available in large multi-database networks and allow the users to deal with a controlled amount of material at a time, while providing more detail as the user looks more closely. To support the process of information elicitation while overcoming the complexity of wide-area information delivery and management, we cannot rely on a collection of indexes which simply contain schema information exported by individual database sources. A more structured and pro-active approach to searching is required. The precursor of such an advanced search approach assumes that we are in a position to impose some logical organization of the distributed information space in such a way that potential relationships between the component database systems in the network can be explored. In addition, to maintain scalability, this must be achieved through a decentralized mechanism which does not proceed via a one step resolution and merging of system information into a single static monolithic structure. These and related issues are addressed herein.
253
This paper is organized as follows. Section 2 presents related work, while section 3 discusses a logical organization for the semantic cross correlation of metadata information from component databases in a multi-database system. Section 4 presents clustering techniques, while section 5 outlines navigation and querying mechanisms. Finally, section 6 presents our conclusions and future work. This work is an extension and elaboration of some early ideas outlined in 14 and 15. In 14 we concentrated on the organization of physical data sharing in large database networks, and described how physical data sharing ties in with a pre-cursor of the conceptual organization of the information space presented in this paper. In 15 we described IR techniques and algorithms used for the physical clustering of databases. In this paper we concentrate on the details of logical database formation, according to subject, based on a common terminology context and present navigation and querying techniques.
2
F i n d i n g Information: A n O v e r v i e w
In this section a number of techniques from different fields for locating information are discussed.
Web-based Resource Discovery The use of the World Wide Web (WWW) has led to the development of a variety of search engines which attempt to locate a large number of WWW documents by indexing large portions of the Web. These tools recursively enumerate hypertext links starting with some known documents. We can classify search engines into two broad categories: centralized index and content-based search engines. Centralized index search engines such as Lycos 11, Web Crawler 19 are manual indexing schemes that rely on techniques which "crawl" the network compiling a master index. The index can then be used as a basis for keyword searches. These systems are not scalable because they use a global indexing strategy, i.e., they attempt to build one central database that indexes everything. Such indexing schemes are rather primitive as they cannot focus their content on a specific topic (or categorize documents for that matter): as the scope of the index coverage expands, indexes succumb to problems of large retrieval sets and problems of cross disciplinary semantic drift. Some of the above limitations are addressed by content-based search engines such as the Content Routing System 23 and Harvest 2. These systems generate summarized descriptions (content labels) of the contents of information servers. The Content Routing System creates and maintains indexes of widely distributed sites. In this distributed information retrieval system a collection of documents is described by means of a content label which in turn can be treated as a document and can be included in another collection. Content labels help users explore large information spaces. However, document collections and their labels are confined to the context of their underlying information servers. Recently, this idea has been extended in the HyPersuit system 26 by generalizing collections so that they may span documents from various servers.
254
The Harvest information discovery and access system 2 provides an integrated set of tools for gathering information from diverse Internet servers. It builds topic-specific content indexes (summaries from distributed information), provides efficient search mechanisms, and caches objects as they are retrieved across the Internet. Each local search engine builds a specialized directory for a certain domain of documents. Federated search engines scan those directories and form federated directories which aggregate documents according to applicationspecific needs.
Subject Gateways A subject gateway, in network-based information access, is defined as a facility that allows easier access to network-based information resources in a defined subject area 9. Subject gateways offer a system consisting of a database and various indexes that can be searched through a Web-based interface. Each entry in the database contains information about a network-based resource, such as a Web page, Web site or document. Advanced gateways provide facilities for enhanced searching. For example the Social Science Information Gateway (SOSIG) 25, incorporates a thesaurus containing social science terminology. This gives users the option of generating alternative terms/keywords with which to search the resource catalog. Another example of an advanced subject gateway is the Organization of Medical Networked Information (OMNI) 16 which allows users to access medical and health-related information. OMNI also facilitates searches across other databases of resources such as databases of dental resources. The key difference between subject gateways and the popular Web search engines, e.g., Alta Vista, lies in the way that these perform indexing. Alta Vista indexes individual pages and not resources. For example, a large document consisting of many Web pages hyperlinked together via a table of contents would be indexed in a random fashion. In contrast this subject gateways, such as OMNI, index at the resource level, thus, describing a resource composed of many Web pages in a much more coherent fashion.
Multi-Database Systems Multi-database (or federated) systems have as their aim the ability to access multiple autonomous databases through querying. The emphasis is on integration and sharing of distributed information and not on information discovery. A particular database may choose to export parts of its schema which are registered in a federal dictionary. A requesting database consults the federal dictionary for existing databases and then imports schema elements that it requires. While this approach might be appealing for a small number of interconnected databases it is clearly not scalable. Locating the right information in a large unstructured network of data dictionaries is extremely cumbersome, has limited potential for success and, more importantly, is error prone as it does not deal with terminology nuances.
255
More recently several research activities in the area have concentrated on the issue of creating semantically enhanced federated database dictionaries 3, 1, 12, 4. Construction of conceptual ontologies on the basis of domain-specific terminologies and formalisms that can be mapped to description logics are also discussed in 8. Some of the issues relating to the identification of semantically related information can be found in 3, where the authors describe an approach that relies on an abstract global data structure to match user terms to the semantically closest available system terms. Concepts grounded on a common dictionary are defined in a domain and schema elements from component databases are manually mapped to these concepts. More recently, a different approach is taken by 7 where a domain-specific classification scheme is built incrementally by considering one schema at a time and mapping its elements in a concept hierarchy. However, both these approaches tend to centralize the search within a single logical index thereby defeating scalability by introducing performance limitations for large networks. 3
System
Organization
In order to improve efficient searching/elicitation of schema information in large multi-database networks, the first task is to partition the multi-database information space into distinct subject (domain-specific) categories meaningful to database users. Categorization and subject classification are common practices in library and information sciences, e.g., the INSPEC indexing and abstracting service covering most of the research literature in Computer Science and Electrical Engineering 22. Domain-specific partitioning organizes databases in logical clusters and makes searches more directed, meaningful and efficient. In addition, a subject directory created as a result of domain-specific database categorization can also provide subject-specific searches and useful browsable organization of inter-component database schema information. There are three basic principles that a system must address to allow for scalable information elicitation. Firstly, an organization of r a w data must be introduced for the discovery of data inter-relationships. Topic classification schemes for this purpose as they summarize related information subspaces together. Secondly, this organizational structure must itself be scalable - that is: interactions with it must be scalable, and maintenance of it must be scalable. Thirdly, users must be presented with a collection of tools (lexicographic, and user friendly graphical interfaces) which allows for easy exploration and interpretation of the information contents of the system. In the following, we address these issues in the context of a logical topic-based architecture for multi-databases.
3.1
Subject-based Database Clustering
Our approach to information elicitation in large database networks relies on logically partitioning the multi-database schema information space into distinct subject (topic) categories meaningful to users. This occurs by creating logical
256 objects called Generic Concepts (GCs) to achieve explicit semantic clustering of associated component database schema elements. Database-content clustering automatically computes sets of related component databases - via their exported meta-data terms - and associates them with an appropriate generic concept, see Figure 1. Generic concepts essentially represent centroids of the inter-component database schema information space - around which databases cluster - and are engineered to describe a particular domain (generic concepts were termed "Global Concepts" in previous work [14]).
Fig. 1. Partitioning a multi-database information space into generic concepts.
To participate in GC-structured database network, a component database must export part of its meta-data to the other databases in the network. This
257
means that the component database administrator must specify which part of the database meta-data can be made available for sharing with other database systems in the network. We refer to these meta-data as the exported meta-data. Figure 1 shows a sample database, called the Universal_Accreditation_Company database, along with a partial representation of its meta-data. Although metadata contain also physical definitions such as definitions of views, ownership, authorization privileges, indexes and access patterns, these (except for authorization privileges) are not important for inclusion in the GC level. A GC organized multi-database schema information space can be viewed as a Web-space that encompasses collections of exported meta-data. A GC organized multi-database schema information space partitions component databases into topically-coherent groups, and presents descriptive term summaries and an extended vocabulary of terms for searching and querying the vastly distributed information space of the component databases that underly it. Databases in this network may connect to more than one GCs if they strongly relate to their content. To circumvent terminology fluctuations we provide a standard vocabulary for interacting with the GCs. In this way we create a concept space (information sub-space) for a specific topic category. The concept space constitutes a type of summarization or synoptic topic knowledge regarding a particular domain, e.g.,
education and training, publications, government tertiary-related departments, etc, and is stored in a GC, see Figure 1. This clustering mechanism results in grouping exported meta-data elements from diverse databases that share important common properties onto a generic concept, associating these properties with the GC representation, and regarding the GC as an atomic unit. A GC is thus a form of a logical object whose purpose is to cross-correlate, collate, and summarize the meta-data descriptions of semantically related network-accessible data. This scheme provides an appropriate frame of reference for both component database schema term indexing and user instigated searches. With this scheme navigation can be considered as browsing through databases exclusively at a topic-level i.e., from topic area to topic area such as from educational training, to publications, government departments and so on. To put the organization of a concept space into perspective, we consider the case of a domain based on educational information provided by a large number of interconnected databases as shown in Figure 1. This figure also illustrates how a component database (Accreditation) - which provides information about accreditation of courses and cross-institutional subjects, various private/public educational training information and other similar or related data - is connected to the GC network. In its original form the Accreditation database, maintains information only on education service providers, their courses, accreditation committee members, accreditation processes and related information. Figure 1 shows the Accreditation database along with a partial representation of its associated meta-data and schema. It also illustrates how this component database may become part of a larger network by establishing weighted links to GCs implementing related areas of interest. Consequently, the Accreditation database is not only able to source appropriate information on its subject matter but also
258
to provide matching information about enrollment programs, training schemes, government programs, research activities and publication data. By linking to a certain GC, databases agree to associate with each other and thus inter-component database organization is achieved implicitly. In addition, GCs are interconnected by weighted links (called content links) to make the searches more directed and meaningful, see Figure 1. Each of the component databases may also link less strongly (e.g., 7/10) to other GCs which have their own associated cluster of database nodes. Presently, the degree of relatedness between GCs is decided by database administrators. Accordingly, a single database, e.g., Universal_Accreditation_Company, may be simultaneously involved in several clusters of databases (information sub-spaces) to varying degrees, as dictated by the weights of its content links to the various GCs. The resulting GC structure forms a massive dynamic network, resembling a cluster-based associative network (a variant of semantic networks that uses numerically weighted similarity links). Overall a networked information system may be viewed in terms of three logical levels. The bottom level (Figure 1) corresponds to the schemas of the component databases. The middle level represents exported meta-data for the database schemas. The top most level corresponds to the concept space (GC) level. This level contains abstract dynamic objects which implement the clustering of related portions of the underlying component meta-data and materialize the GCs in an object-oriented form. Figure 1 illustrates that there is a one-to-one correspondence between database schemas and their meta-data representations, while an entire collection of exported meta-data corresponds to a single concept-space. This three-tier architecture is the key ingredient to information elicitation in distributed, scalable systems. It provides the ability to describe varying levels of aggregated database sources and the granularity of the information components, i.e., exported meta-data terms, that comprise them. It generates a semantic hierarchy for database schema terms in layers of increasing semantic detail (i.e., from the name of a term contained in a database schema, to its structural description in the meta-data level, and finally to the concept space level where the entire semantic context - as well as patterns of usage of a term can be found). Searches always target the richest semantic level, viz. GC level, and percolate to the schema level in order to provide access to the contents of a component database, see section 5. This type of content-based clustering of the searchable information space provides convenient abstraction demarcators for both the users and the system to make their searches more targeted, scalable and effective. This methodology results in a simplification of the way that information pertaining to a large number of interrelated database schemas can be viewed and more importantly it achieves a form of global visibility 17. Although GCs provide synoptic information about their underlying database clusters, they do not require integration of the data sources. This approach comes in strong contrast with approaches to semantic interoperability based on explicit integration of conceptual schemas on the basis of semantic lexica 3, 4. The advantage of forming conceptual
259
database clusters is that searches are goal-driven3 and the number of potential inter-database interactions is restricted substantially as it facilitates the distribution and balancing of resources via appropriate allocation to the various database partitions.
3.2
Generic Concept Characteristics
Individual GCs are useful for browsing and searching large database collections because they organize the information space. For example, the Education and Training Providers concept space provides a common terminology basis upon which database nodes dealing with enrollments, courses, training, accreditation, etc, (see Figure 1), achieve knowledge of each others information content. A GC is a definitional or schematic construct: it corresponds to a class hierarchy depicting all terms within the topic sampled by the GC. The GC structure is illustrated in Figure 2. This figure shows that each GC is characterized by its name and the context of its terms (term hierarchy and term descriptions) for each specific topic. Terms within a GC are shown to have a distinct meaning (sense) and context. This concept space consists of abstract descriptions of terms in the domain, term senses, relationships between these terms, composition of terms, terminology descriptions, hypernym, hyponym, antonyms-of, part-of, member-of (and the inverses), pertains-to relations, contextual usage (narrative descriptions), a list of keywords, and other domain specific information, that apply to the entire collection of members of a GC, Figure 2. Hence, the GC structure is akin to an associative thesaurus and on-line lexicon (created automatically for each topic category). Thesaurus-assisted explanations created for each subject-based abstraction (GC-based information subspace) serve as a means of disambiguating term meanings, and addressing terminology and semantic problems. Therefore, the GC assists the user to find where a specific term that the user has requested lies in its conceptual space and allows users to pick other term descriptions semantically related to the requested term. Operations on a GC object include mapping services which map GC provided terms to semantically related terms in the component databases. They also include summarization services which summarize the exported meta-data from component databases to implement a GC. Summarization services aggregate networks of exported meta-data terms (one per component database). This mechanism is described in a later section. An example of the GUI for some of the the terms included in the educational GC is given in Figure 3. Here, we assume that a user who searches the entries in the educational GC is interested in the term course and wishes to gain more insight into its semantic context. The first step after entering the term is to choose the s e n s e s from the list the GC lexicographic substrate provides. The sense number returned is then associated with the term (as is the case with all other words in the term description). For example, Figure 3 shows that the 3 A goal-driven search accepts a high-level request indicating what a user requires and is responsible for deciding where and how to satisfy it.
260
Fig. 2. Generic concept structure.
term course has eight senses (meanings), but once the domain of discourse is limited to study (education), then only one of the eight can occur. Figure 4 which is an expansion of the specific term chosen, shows how the GC provides the necessary information needed for the contextual representation, i.e., meaning, of a specific term. Other factors such as the context of usage (not shown here due to space limitations) can be combined with its contextual representation to restrict the search space. Thus the user gets a complete picture regarding the semantic context of this and associated terms (see Figure 4) and is free to pick up a desired term(s) which would eventually lead him/her to candidate component data sources. Term entries in this GUI are mapped by means of the mapping services of a GC to the relevant schema terms found in component databases (in the same GC). Information contained in the GCs is stored in an information-repository that resides at a concept server associated with and accessible by the databases clustered around a specific conceptual information space (GC), see Figure 1. The concept server implements an individual GC, performing abstraction and summarization operations on its underlying meta-data. This information-repository contains thus a rich domain model that enables describing properties of the database sources clustered around a GC. 4
Representation
and Clustering
of Schema
Meta-data
In the following we describe a general methodology that aids in clustering databases and creating their corresponding generic concepts. Key criteria that have guided this methodology are: scalability, design simplicity and easy to use structuring mechanisms.
261
Fig. 3. Choosing the meaning of the term course.
4.1
Describing the Meta-Data
Content of a Database Node
In order to initialy cluster component databases a high level description of the m e t a - d a t a content of a database must first be developed. To demonstrate this consider the previous example of the UniversaLAccreditation database, which deals with academic institutions and accreditation processes. This database contains entities such as courses, committees, (accreditation) processes, etc. We use a variant of an information retrieval (IR) technique called, star technique, where a t e r m is selected and then all terms related to it are placed in a class[10]. Terms not yet in a class are selected as new seeds until all terms are assigned to a class. The variant of the star technique that we are using starts with a t e r m represented as an abstract class (term descriptor class), then an additional t e r m t h a t is related to the term selected is represented as a another class and is connected to the selected term. The new term is then selected as a pivot and the process is repeated until no new terms can be added. In this way a context graph created for a specific database schema. For example, the context graph for the Universal_Accreditation component database (Figure 5) contains nodes
262
Fig. 4. More contextual information regarding the term course.
which correspond to the abstract term descriptor classes committee, institutions, courses etc., while the context graph edges depict inter-connections (association, generalization, specialization or containment) between the terms within this particular database. Term interrelations are determined on the basis of a reference lexicographic substrate that underlies all the GCs in the network. For this purpose we use the lexicographic system 4 WordNet [13] that supports semantic term matching through the use of an extensive network of word meanings of terms connected by a variety of textual and semantic relations. To facilitate clustering and discovery of information, we require that a component database (e.g., Universal_Accreditation) can be totally described in terms of three sections which contain a synoptic description of the meta-data content of the database; associations between meta-data terms in the form of a semantic4 This lexicographic tool is presently used only for experimental purposes and will be replaced by an appropriate subject gateway in the near future.
263
Fig. 5. Describing a component database.
net; and finally, links from these descriptions to other related databases in the network. This information can be viewed by users of the system once they have chosen a component database that potentially matches their interests (see section 5). Figure 5 illustrates that each database node contains the following sections: a feature descriptions, a context graph, and a GC connections section. The feature descriptions section contains information about terms, composition of terms, remarks about the meaning of terms, hypernym, hyponym, antonyms-of, part-of, member-of (and the inverses), pertains-to relations and lists of keywords. This section may also include certain details such as: geographical location, access authorization and usage roles, explanations regarding corporate term usage and definitions, domains of applicability and so on. The feature descriptions entries are partially generated on the basis of WordNet and contain information in the form represented in Figures 2, 3 and 4. The context graph section contains a non-directed graph which connects term synopses (in the form of term descriptor classes) found in the iJniversaLAccreditation database schema. Except for viewing purposes, the term descriptor nodes and their link structure are used
264 in the clustering of databases to form the generic concepts. Each of the term descriptor nodes defines (in conjunction with its respective entry in the feature descriptions window) a common structured vocabulary of terms - describing the term in question, e.g., course, - and a specification of term relationships within that particular subject. Finally, the GC connection section shows how the Universal_Accreditation database is related, i.e., content link weights, to other GCs in the network.
4.2
Similarity-based Clustering of Database Nodes
Similarity-based clustering of database schemas organizes databases into related groups based on the terms (term descriptor nodes) they contain and the link structure of their context graphs. Our clustering algorithm determines the similarity between two graphs (representing two different database schema meta-data) based on both term similarity and link similarity factors. This is accomplished in two steps. Firstly, a pairwise-similarity of nodes in two context graphs is computed. From this an initial "pairing" of the nodes is determined. In the second step a comparison of the link structure of two context graphs is made based on the inter-node pairings and a semantic distance value is calculated. We chose this term/link similarity-based algorithm because it is relatively easy to implement and avoids generating very large clusters.
Term-based Similarity: this is calculated using cluster analysis techniques 5 to identify co-occurrence probabilities - representing the degree of similarity - between two discrete terms. Our similarity metric is based on the meaning of the collection of terms representing the topical context (viz. semanticlevels) of a particular term, e.g., course, and the synonyms of these, see Figure 3. The comparison is based on: a conversion of each context graph node (e.g., term descriptor) Committee, Process, Subject, Course, etc. (see Figure 5) to a corresponding matrix of noun terms (containing the entire topical context of a term); and a subsequent comparison of terms within these matrixes. A matrix an,m of (noun) terms, representing the topical context of a particular term, a~,l (course say), will correspond to the name of the term descriptor in the context graph. The synonyms of this term will be ai,2, ai,3 ... ai,m (course-of-study, course-of-lectures). Terms ai-x,j (X > 0), e.g., education, educational-activity, will be more general than terms a i j , while terms ai+x,j will be more specific, e.g., CS-course. In the final step, all synonyms for these terms are generated to produce the node's a complete topical description matrix an,m for a specific term. Similarity analysis is mainly based on statistical co-occurrences of term descriptor objects based on techniques which has been successfully used for automatic thesaurus generation of textual databases 5, 21. In fact we base our term-based similarity on the improved cosine formula 21 which is used to calculate the semantic distance between the vector for an item in a
265
hierarchical thesaurus and the vector for a query item. To provide the right ontological context for semantic term matching, we use again the massive semantic net WordNet [13]. C o m p a r i s o n of t h e c o n c e p t u a l s t r u c t u r e of two c o n t e x t graphs: to determine the structural and semantic similarity between two graphs, we based our algorithms regarding conceptual similarity between terms on heuristicsguided spreading activation algorithms, and on work in the information retrieval area presented in [20]. These approaches take advantage of the semantics in a hierarchical thesaurus representing relationships between index terms. The algorithms calculate the conceptual closeness between two index terms, interpreting the conceptual distance between two terms as the topological distance of the two terms in the hierarchical thesaurus. During this process similarity between nodes (term descriptors) is established by considering the edges separating the nodes in the context graph as well as the actual graph structure. Some early results regarding the comparison and clustering process are described in [15].
Once similarity between nodes has been established context graphs are aggregated to create GCs. The aggregation of the context graphs from various component databases, results in the clustering of inter-related database schemas, see
266
Figure 6. The aggregation algorithm employed does not integrate the aggregates, as is the usual case with other approaches 8, but rather links descriptor classes at the GC level with corresponding term descriptor classes in its underlying cluster of database context graphs. Again this association is performed on the basis of the reference lexicographic substrate (WorNet). For each database cluster, a GC is created to represent the area of interest (or concept) that the group embodies, e.g., Education and Training Providers GC for the Employee Training, Accreditation, and Government Education Center databases as depicted in Figure 2. 5
Schema
Term Navigation
and Querying
Information elicitation spans a spectrum of activities ranging from a search for a specific data-item(s) (contained in possibly several component databases) to a non-specific desire to understand what information is available in these databases and the nature of this information.
5.1
Navigation Techniques
There are two basic modes in which searching of the system may be organized. These search modes depend upon the nature of the information a user is attempting to access, and how this information relates to the database that user is operating from. Serendipity, exploration and contextualization are supported by means of indexing based upon terms contained in the component database context graphs. In such cases the user is interested in finding out about a particular topic rather than a specific information (schema) item. We call this former form of exploration index-driven. Alternatively, if a user is seeking data which is closely related or allied to her/his local database, then searching may be organized around the weights of content links of this database to other GCs in the network. We refer to this form of exploration as concept-driven. Conceptdriven querying is the subject of a previous publication 18. In this paper we will concentrate on index-driven exploration and on the querying of schema-related information. Index-driven navigation allows the users to deal with a controlled amount of material at a time, while providing more detail as the user looks more closely and is related to the dynamic indexing schemes and incremental discovery of information requirements for information elicitation. In order to traverse the index a user will have to decide on a number of key request terms, and then select synonyms or more general (and perhaps more specific) derivatives of these key terms. The resulting query structure - generated on the basis of terms extracted from WordNet entries - can then be compared against the context graph structure of component databases. User specified term comparison starts at the top of the GC generated index and gradually percolates down to the required level of specificity by following the terms at each level. Figure 7 depicts this process in terms of a user query
267
Fig. 7. Accessing the index.
requesting information about courses a t various institutions. Here we assume that the user has already specified that s/he is interested in the contents of the Education & Training GC. The graph of the user's query supplied terms contains a node Course and this term is used to traverse the GC generated index and arrive at the collection of databases which include this term (or its aliases) in their own descriptions. The index-driven navigation process starts with the most general terms possible, e.g., act, human activity, that correspond to the requested query term (course). These terms are generated by the GC (via the WordNet) and are presented to the user for selection. Once the user has selected a general term, most specific terms are revealed, e.g., education. Once a GC term matching a user supplied term is selected, a link is established with the context graphs of all component databases containing the desired term (or its aliases). In this way the user can obtain contextual information and possibly a partial view of potentially matching databases and then s/he can decide whether a candidate database is useful or not. This hierarchical form of schema term navigation guarantees that a user supplied term correlates semantically with the content of the component databases underlying a GC cluster. The process is then repeated for all the other
268
terms in the user's query graph (i.e. the remaining unlabeled nodes in Figure 7). Thus, by matching the user query graph nodes to semantically equivalent GC terms, we can infer a number of component databases that are most closely associated to the user query.
5.2
Querying of Domain Meta-Data
When the user needs to further explore the search target, intensional, or schema queries 17 - which return meta-data terms from selected schema terms - can be posed to further restrict the information space and clarify the meaning of the information items under exploration. Such domain-specific queries should not be confused with queries which target the data content of the component databases (to which we refer to as distributed queries/transactions). Intensional queries are particularly useful for assisting users who are unfamiliar with the vocabulary of terms that can be used in connection with distributed queries/transactions or with the range of information that is available for responding to distributed queries. Sample intensional queries related to the GC in Figure 4 may include the following:
query- 1: Find the set of common super-terms of course. query-2: Find all terms more specific than course and all their parts under sense education.
query-3: Find the smallest common super-term of course of lectures and workshop.
query-4: Find all parts of the term course. query-5: Which are the common properties of refresher course and seminar? que.ry-6: Find all terms which contain the properties lesson and classroom project.
query-'/: What is the definition of the term refresher course? All of the above queries - except for the last one - are rather intuitive. The last query returns a narrative description of the requested term in English (if available). Finally, when users feel sufficiently informed about the contents and structure of component database schema terms they have explored, they can pose meaningful distributed database requests which target the data content of the relevant component databases. 6
Experimentation
The framework that we described in this paper is being implemented on Sun SparcStations under Solaris 2 using GNU C + + and CGI scripts. In order to evaluate automated clustering a test platform based on the clustering of about 100 networked databases has been created. There are two basic areas of experimentation being pursued. Firstly, there is the question of how well the initial automatic clustering of databases based on each component databases description
269 can be performed. That is, the scalability question of finding appropriate initial relationships in the presence of large numbers of information sources. The types of experiments performed here are somewhat allied with the field of information retrieval and clustering. The second set of experiments, on the other hand, deals with the processing and communications necessary to support the underlying distributed structure by which the generic concepts and their inter-relationships are implemented, queried and updated. This second group of experiments thus has its roots in the fields of distributed/parallel processing and communications performance. In a similar vein to IR experiments, the first set of experiments are based on the notion of retrieval and accuracy (as defined within IR). To achieve this, a collection of a hundred relational databases has been procured from a large organization's collection of information systems. A manual clustering of these was then performed by a domain "expert" who had full intimate knowledge of the organization's environment. This clustering was essentially based on where each database fitted into the various departments within the organization, and how these departments interacted/overlapped - the latter being identified via analysis of database table usage within the various departments. Thus, we clustered databases based on the actual usage of data from the various information components as dictated by the organization of the environment that the databases were set up to model in the first place - but in a macro (organization wide) sense rather than a micro (department based) sense. Experiments have been performed (and continue to be performed) to: 1. identify if automatic clustering can achieve a "near perfect" initial organization of the database collection - or at least be statistically significantly better than "raw" automatic clustering, which involves the identification of an appropriate heuristic for measuring the similarity between database descriptions; 2. compare results against other standard automatic clustering packages (e.g., those found in IR); 3. determine what set of descriptive "primitives" are essential (and minimal) to achieve a satisfactory degree of clustering; 4. determine the "robustness" of the description process - i.e., give some indication of how much variation there can be within a description before the automatic clustering becomes unsatisfactory. This last experiment is important as it must be remembered that different people may be responsible for the construction of different database descriptions. Thus, the automatic clustering must be relatively robust in terms of the way different people may describe the same object. It is expected that, given all descriptions will be generated using the same thesaurus, the system should prove relatively good at detecting differing descriptions of a single object. Currently, experiments have been performed using a "full" database description involving the synonyms, generalizations and terms senses, as well as the structural relationships between these terms, see Figure 4. Initialy, the term
270
matching component was based on the standard similarity metric proposed by Dice 5, and the structural similarity was based on the notion of spreading activation energy 15. It was found, however, that the accuracy and retrieval of this particular approach was not significantly better than the clustering of the "raw" database descriptions using Dice's method directly. Upon analysis it was discovered that performance was degraded due to the un-directed nature of the context graph. Thus, in a subsequent set of preliminary experiments, the notion of spreading activation energy was dropped, and a ranking of similarity based on the hierarchy of the graph was introduced. This resulted in a huge improvement in the retrieval and similarity figures which indicated the automatic clustering to be significantly better than the base-line clustering.
7
S u m m a r y and Future W o r k
This paper described the fundamental aspects of a scalable, semantically oriented, configurable distributed information infrastructure that supports information discovery and retrieval across subject domains in networked databases. The proposed logical architecture extracts semantics from database schemas and creates dynamic clusters of databases centered around common topics interest (viz the generic concepts). Large-scale searching is guided by a combination of lexical, structural and semantic aspects of schema terms in order to reveal more meaning both about the contents of a requested information item and about its placement within a given database context. To surmount semantic-drifts, the terminology problem and enhance database retrieval, alternative search terms and term senses are suggested to users. This architecture enables users to gather and rearrange information from multiple networked databases in an intuitive and easily understandable manner. Experience with this configuration suggests the clustering mechanisms used provide a valuable discovery service to end users, and that the logical organization used supports the ability of the system to scale with modest increases in GC label sizes. Future work addresses the semi-automatic generation of link weights based on term co-occurrences using statistical/probabilistic algorithms. In IR these algorithms use word and/or phrase frequency to match queries with terms 5. In the current prototype link weights are established at a clustering phase on a tentative basis only. However, it is expected that during execution link weights to GCs may need to be updated (strengthened or weakened) over time depending on interaction, new GCs may be formed, and existing GCs may need to merge. The next suite of experiments to be performed will deal with the characteristics of the link weight update and GC split/merge processes. From this policies will be developed (e.g. delayed/batch updating of GC information), and then evaluated.
References 1. Arens Y., et al. "Retrieving and Integrating Data from Multiple Information Sources", Int'l Journal of Cooperative Information Systems, 2, 2, (1993).
271
2. Bowman. C. M., et al. "Harvest: A Scalable, Customizable Discovery and Access System", Univ. of Colorado - Boulder, CS Dept., techn, report CU-CS 732-94, (1995). 3. Bright M., Hurson A., Pakzad S. "Automated Resolution of Semantic Heterogeneity in Multidatabases" ACM ToDS, 19, 2, (1994). 4. Castano S., De Antonellis V. "Semantic Dictionary Design for Database Interoperability", 13th Int'l Conf. on Data Engineering, Birmingham, April (1997), 43-54. 5. Everitt B. "Cluster Analysis", Heinemann Educational Books Ltd., Great Britain, (1981). 6. Kahle B., Medlar A. "An Information System for Corporate Users: Wide Area Information Servers", The InteroperabilityReport, 5, III, (1991). 7. Kahng J., McLeod D. "Dynamic ClassificationalOntologies: Mediation of Information Sharing in Cooperative Federated Database Systems", in Cooperative Information Systems: Trends and Directions, Papazoglou M. P., Schlageter G. (eds), Academic-Press (1997) 179-203. 8. Kashyap V., Sheth A. "Semantic Heterogeneity in Global Information Systems: the Role of Metadata, Context and Ontologies", in Cooperative Information Systems: Trends and Directions, Papazoglou M. P., Schlageter G. (eds), Academic-Press (1997) 139-178. 9. Kirriemuir J. et al., "Cross-Searching Subject Gateways", D-Lib Magazine, January (1998). 10. Kowalski G. "Information Retrieval Systems: Theory and Implementation", Kluwer Academic Publishers, (1997). 11. Manldin L.M., Levitt J.R. "Web-agent related Research at the CMT", Procs. ACM Special Interest Group on Networked Information Discovery and retrieval: SIGIR '9~, August (1994). 12. McLeod D., Si A. "The Design and Experimental Evaluation of an Information Discovery Mechanism for Networks of Autonomous Database Systems", 11th Int'l Conf. on Data Engineering, Taiwan, Feb. (1995) 15-24. 13. Miller G. "WordNet: A Lexical Database for English", Communications of ACM, 38, 11, Nov. (1995). 14. Milliner S., Bouguettaya A., Papazoglou M.P. "A Scalable Architecture for Autonomous Heterogeneous Database Interactions", 21 Int'l Conference on Very Large Databases, Zurich, Switzerland, Sept. (1995). 15. Milliner S., Papazoglou M., Weigand H. "Linguistic Tool based Information Elicitation in Large Heterogeneous Database Networks", NLDB '96 Natural Language and Databases Workshop, Amsterdam, June (1996). 16. "OMNI, Organizing Medical Networked Information", http://omni.ac.uk/ 17. Papazoglou M.P. "Unraveling the Semantics of Conceptual Schemas", Communications of ACM, 38, 9, Sept. (1995). 18. Papazoglou M.P., Milliner S. "Pro-active Information Elicitation in Wide-area Information Networks", Procs. of the Int'l Symposium on Cooperative Database Systems for Advanced Applications, World Scientific, Japan, Dec. (1996). 19. Pinkerton B. "Finding what People Want: Experiences with the WebCrawler", Procs. 1st Int'l Conference on the WWW, Geneva, May (1994). 20. Rada R., Bicknell E. "Ranking Documents Based on a Thesaurus", Journal of the American Society for Information Science, 40, 5, May (1989). 21. Salton G.E, Buckley C. "Term-Weighting Approaches in Automatic Text Retrieval", Information Retrieval and Management, 24, 5, (1988), 513-523.
272
22. Schatz R.B., et. al "Interactive Term Suggestion for Users of Digital Libraries", 1st ACM International Conf. on Digital Libraries, Bethesda MD, March (1996), 126-133. 23. Sheldon M.A. "Content Routing: A Scalable Architecture for Network-Based Information Discovery", PhD thesis, MIT, Dec. (1995). 24. Sheth A., Larson P. "Federated Database Systems for Managing Distributed, Heterogeneous and Autonomous Databases". Computing Surveys, 22, 3, Sept (1990). 25. "SOSIG: The Social Science Information Gateway", http://www.sosig.ac.uk/ 26. Wiess R., et al. "HyPersuit: A Hierarchical Network search Engine that Exploits Content-link Hypertext Clustering", 7th ACM Conf. on Hypertext, Washington DC., March (1996).
MUSE - An Interactive Networked Multimedia Applications Specification Environment with E-LOTOS Translator Luciano Paschoal Gaspary Maria Janilce B. Almeida Universidade Federal do Rio Grande do Sul Instituto de Informfitica Curso de P6s-Gradua~o em Ci~ncia da Computa~ao Campus do Vale, Bloco IV - Bento Gon~alves, 9500 - Agronomia - 91591-970 Porto Alegre, RS - Brazil E-mall: {paschoal, janilce} @inf.ufrgs.br Abstract. This work presents MUSE, a graphical environment for modeling interactive networked multimedia applications. Through an advanced graphic interface and a new highlevel authoring model, it is possible to create complex systems in a fast and intuitive way. The authoring model proposed in this work and adopted by the environment deals with media objects distributed in a computer network, allowing the definition of acceptable presentation delay thresholds and alternative media objects. Due to the large expressiveness of the model, however, specifications with logical and temporal inconsistencies may be generated. For this reason, the tool also provides E-LOTOS specifications, which may be used to analyze and verify the temporal requirements defined by the author.
I
Introduction
The 90's have been known by the use of multimedia applications in several fields of the human activity such as education, medicine and entertainment. These applications have become increasingly sophisticated along the time, and nowadays they are executed in distributed environments, operating transparently in heterogeneous platforms. The possibility of having an application with its media objects dispersed in a network influences the creation and modeling of such applications. Users must provide the authoring tools with information like temporal restrictions, defining acceptable delay thresholds to the presentation of the elements that compose the system and establishing the presentation of alternative media objects. The definition of these restrictions is accomplished based on a synchronization model, which dictates the rules about how the media objects of an application can be related in time. Several synchronization models have been proposed 1. Most of them are both flexible and very expressive. That is the reason why the resulting specifications can be source of incoherences, where the logical and temporal consistency of the involved media objects can not be assured. An alternative would be to use directly a formal description technique (FDT) to describe the applications, making its analysis possible and so guaranteeing its consistency. The disadvantage of this direct usage, however, is the high complexity inherent to FDTs. So, the need of
274
having a structured high-level model to specify interactive networked multimedia applications becomes evident. The resulting specifications shall then be translated to an FDT, so that verification and simulation methods can be applied to them. In this context, an interactive networked multimedia applications authoring model was created. MUSE (MUltimedia Applications Specification Environment) was developed to support this model, allowing the user to easily define a multimedia presentation according to the MHEG-5 standard 2. The adoption of MHEG-5 allows multimedia information to be shared without worrying about the platform or operating system used, providing specification and development of portable applications. To make the validation process of the specifications possible, the environment automatically generates E-LOTOS specifications. This work is part of DAMD (Distributed Multimedia Applications Design) project, sponsored by the Brazilian research council. Its main objectives are to provide a methodology to completely cover the distributed multimedia applications development cycle and to allow authors who are not expert in formal methods to easily develop their applications. The project was developed according to figure 1. MUSE, in a certain way, centralizes the process that comprehends modeling and presentation of applications. Specifications created by the user are validated and the obtained results are presented to him in a quite readable way in the own tool. The specification-validation process repeats until the incoherences are eliminated. After that, MHEG-5 applications are generated and can be executed by the engine. modeling process presentation user
HHEG5 Engine
specifications
J
Simulation/Verifi~Uon~i
Fig. 1. Structure of the DAMD project This paper is organized as follows: section 2 presents important aspects to be considered in the applications authoring process, relating them to some multimedia synchronization models pointed by the literature. This section also presents the proposed authoring model. In section 3 basic aspects of the E-LOTOS FDT are presented, as well as a mechanism to represent specifications generated by the authoring model in this formal technique. Section 4 illustrates the functionality of the environment and in section 5, one can read the final considerations.
2 Proposed Authoring Model The specification of multimedia applications is accomplished with base on three fundamental aspects: logical structuring, establishment of temporal relationships and spatial definition among the elements belonging to the application. The logical
275
structuring is concerned to offer abstraction mechanisms, providing a wide and structural view of the application. The specification of the temporal behavior involves the definition of synchronization relations among media objects. The spatial synchronization cares about adjusting the positioning of the visible media objects according to the output devices (video). The temporal relations are established according to a synchronization model, which imposes rules on how these elements can relate to each other. Several models have been proposed in the literature. One of the most adopted by existent authoring tools is the time-line based one 3. However, it presents many limitations such as the difficulty both to modularize the application and to establish relations among elements with variable or unknown duration like user interaction 4. The hierarchical model also presents deficiencies. The most important one is that the construction and reading process of the specifications is not natural. It is not clear the order in which the media objects will be presented. Besides, the model does not allow the establishment of some synchronization rules 1, which restricts the expression power of this model. Models based on references to points are not adequate to model distributed multimedia applications, because there is no an explicit time notion. Thus, temporal restrictions can not be expressed and temporal irregularities (common in distributed systems) are ignored in this model. In synchronization models based on Petri nets, it is possible to specify most of the important synchronization rules required for modeling multimedia applications 5. Among the models up to now presented, this one provides the largest expression power and flexibility. Moreover, as Petri net is a formal model, it makes applications analysis possible, allowing its consistency to be guaranteed. Its largest disadvantage, however, is its complexity; the manipulation of large specifications may become difficult because of the state explosion problem. In this work, an authoring model that joins mechanisms for logical structuring the applications to a synchronization model similar to HTSPN is proposed. The logical structuring level is based on the concept of scenes and groups, providing a broad view of the application. The definition of temporal synchronizations is done in each scene by means of a simplified graph. The spatial synchronization allows media objects to be positioned considering the output device (see figure 2).
2.1 Logical Structuring The complexity of multimedia applications increase according to the growth of the number of involved media objects and, consequently, to the several temporal relationships established among them. This is the fundamental reason why the specification of these applications in only one plane is inappropriate. To solve this problem, the concept of scenes was incorporated into the model considering the MHEG-5 standard. Multimedia applications can be organized as a group of scenes related by events, which provide the navigation among them. Each of these scenes can be seen as a black box with an internal behavior that, under certain conditions, enables the presentation of other scenes. The use of this concept, however, does not solve completely the problem of complexity, since a specification with many scenes will be hardly understood. Trying
276
to make easier the understanding of so large applications, a hierarchy mechanism was added to the model through the concept of group of scenes. The top of figure 2 illustrates the logical structure of an application, composed by four scenes (Scene l, Scene2, Scene3 and Scene4). Three of them (Scene2, Scene3 and Scene4), due to the cohesion established among them, were gathered in Groupl. The arcs that link groups and scenes in the logical structure do not represent temporal synchronizations, but choices. For example, a scene A tied up to two scenes B and C indicate that the temporal behavior of the scene provides two ways for the application to evolve: either to B or to C, only depending on the dynamic behavior of the application. This evolution is materialized by the use of the transition icon, to be mentioned in the following section.
Fig. 2. Structure of an interactive multimedia application Usually, there are media objects whose presentation embraces several scenes. With the purpose of increasing the expressiveness of the model, the possibility of representing media objects shared by several scenes was created. Figure 3 shows an application organized in three scenes (Scenel, Scene5 and Scene6) and a group (Groupl). The image Logo is presented simultaneously to the whole application, and Instructions, during the presentation of Groupl and Scene5.
277
Fig. 3. Media objects shared among scenes and groups From the authoring process perspective, the proposed structure facilitates the reuse of scenes and groups that repeat in different specifications. Besides, the model allows the development of templates - basic pre-constructed scenes - whose utilization makes the specification process evolving and incremental. One can have a set of templates, so that the specification process, in this case, is reduced to joining these different scenes, lessening drastically the development efforts.
2.2 Temporal Synchronization The temporal synchronization of an application, as mentioned previously, refers to the ordering of the presentation of its media objects in time. Each media object has a presentation duration that may or may not be foreseen, depending on its nature. The following topics present how the synchronization relationships can be established.
Basic Synchronization.
Media objects can be presented sequentially or simultaneously. In the sequential presentation, the playout of a media object depends on the end of another's. In figure 2 both types of basic synchronization appear. In Scene1, the presentation of a text (Intro) is followed by the presentation of an image (Machines). In Scene4, there is the simultaneous presentation of a video (Video1) and a button (Button).
Event Duration and Delay. A minimum and a maximum duration of presentation are associated to each media object. In the case of an image or a text, these values are equivalent because they are time-independent media objects. When one deal with media objects like audio and video, however, it is important to determine both a minimum and a maximum presentation duration, since these media objects will be hardly presented at the nominal rate due to problems like network traffic. The representation of these durations is given by an interval. To make the modeling of a delay between the presentation of two consecutive media objects possible, a special icon can be used. It does not have any media object associated to itself but only a specific value representing how long it has to wait to start the presentation of its successive media. Figure 4 illustrates three slides (Slidel, Slide2 and Slide3) being presented sequentially with a delay of three seconds between the first and the second and a delay of five time units between the second and the third one.
278
Fig. 4. The delay icon
User Interaction and Scene Transition. User interaction corresponds, for instance, to a button click or an object selection. It is represented in this model as a constructor whose presentation duration is uncertain, varying between the minimum and maximum values associated to it. When the maximum threshold is reached, the scene continues with its presentation. It is still possible to specify a button without maximum duration; in this case, its evolution will only happen after the interaction. The user interference is normally associated to a scene transition. Transition is the constructor that makes the navigation among scenes possible. Its execution involves both the immediate suspension of the presentation of all the media objects belonging to the current scene and the beginning of a new scene presentation. In Scene4 (see figure 2), the transition to Scene3 occurs after the hitting of the button; if the video (Videol) is still being presented at that instant, it is interrupted. The connections described in the logical structure and the transitions used in the temporal structure must be consistent to each other. In Scenel, for example, the only acceptable transition is to Groupl, once in the logical structure the scene only has connection to the icon that indicates this group. Synchronization Points. Synchronization points allow the beginning of the presentation of one or more media objects to be associated to different policies related to the end of the presentation of other media objects that converge to these points. To simplify the graphical representation of the authoring model, synchronization points involving only two presentations are not shown. For instance, in figure 4 the synchronization points between Slidel and the delay and between the delay and Slide2 are not presented.
Fig. 5. Synchronization point and firing rules
279
To increase the specification power, the model has adopted some policies widely commented in the literature. They allow the association of different behaviors to the synchronization points [6]. For simplification, the model only supports three of them: the synchronization point is fired when the presentation of a master media object is finished, interrupting all the others. This rule could be used in the example of figure 5a above if one wishes that the end of the presentation of Video (master) causes the interruption of Audio l, starting Audio2 (see figure 5b). The master media object is identified by the presence of the character m or the word m a s t e r close to it.
-
Master:
-
Earliest:
-
Latest:
the synchronization point is fired when the presentation of the first media object is finished, resulting in the interruption of the others. This rule is graphically represented by the presence of the character e or the word e a r l i e s t close to the synchronization point. the absence of an indication close to the media object or to the synchronization point means that all the media objects that precede this point will be executed (or they will conclude due to the elapsing of their maximum presentation duration) before the synchronization point is fired (figure 5a).
Instants of Synchronization. In MUSE, the synchronization among media objects in other instants than the beginning and end of its presentations requires the division of these media objects in parts, creating a set of segments. The granularity of this division is associated to the precision degree desired for the synchronization. Figure 6 shows the synchronization of two subtitles (Subtitlel and Subtitle2) with a video (VD), where the latter is divided into four segments. The first subtitle is presented simultaneously to the second video segment and the second subtitle together with the third segment.
Fig. 6. Synchronization of a video with subtitles 2.3 Spatial Synchronization The spatial synchronization allows the author to visualize the positioning of the visible media objects of a scene. It is not possible to accomplish the spatial structuring considering a certain time elapsed after the beginning of the scene execution. It is so because each of the executions of the application, due to the acceptable temporal variations, the media objects can be presented in different instants. For this reason, the spatial synchronization is always accomplished with relation to the presentation of a media object. The spatial arrangement of the media objects of Scenel (see figure2)
280
during the presentation of Machines, for example, will only allow the bitmap Machines to be organized. On the other hand, the spatial view of Scene4 during the presentation of Videol will present the media objects Videol and Button. The appearance of Button occurs because it is defined to be simultaneously presented with Videol.
2.4 Example of Model Usage The example illustrated in figure 7 models the application proposed in [1], where initially a video (VD) and an audio (AU) are executed simultaneously. Following, a recorded user interaction (RI), a sequence of three slides (P1-P3) and an animation (ANI) which is partially commented by an audio sequence (Audio2) are presented sequentially. During the animation, a multiple-choice question is presented to the user (Interaction). If the user makes the selection, a final image (P4) is presented. This is just one of several ways of representing this application. The ease in understanding it is obtained mainly by the user's good sense in the moment of its specification.
Fig. 7. A simple example of the model usage
281
3
Representation of Multimedia Applications in E-LOTOS
The formalization of specifications is important for the process of their validation. The proposed authoring model, due to its high flexibility and expressiveness, allows both temporally and logically incoherent specifications to be defined. The analysis process detects, for example, conflicts in resources usage and tests if the application's end can be reached from all the possible navigation paths. Thus, specifications described by an author according to the model presented in the previous section are translated to a formal representation, analyzed and the obtained results are presented to the user, who will make the necessary adjustments. The formal description technique E-LOTOS (Enhancements to LOTOS) 7 is an enhanced version of LOTOS and is in standardization process. The main innovation of the language is the incorporation of quantitative time notion, allowing the definition of instants in which actions or events may happen. This is a fundamental feature for representing multimedia applications and, for this reason, E-LOTOS was chosen to formally represent them. The representation of multimedia applications is hierarchical and considers the four essential elements of the authoring model: application, group, scene and media object. All these elements are modeled as processes that evolve according to previously established synchronization relationships. The way of formally represent multimedia applications commented in this section is based on the approach presented in 8. Further details are presented in the following topics.
3.1 Data Representation and Root Process Instantiation Data representation is done by means of a library called classes, which define data types for all possible media objects. There are types like BitmapClass, StreamClass and SwitchButtonClass, whose definition is based on their respective MHEG-5 classes. For example, the fields of BitmapClass are the media object, its position in the output device and its dimensions. The application is started from the instantiation of the root group process. After that, the application is indeed able to evolve.
3.2 GroupRepresentation In the representation of groups, the hiding operator is used. Taking the example of figure 8, one can see that some internal events like the beginning of both Scene2 (s_Scene2) and Scene3 (s_Scene3) are not visible outside the process (1). These events are used to synchronize the presentation of the scenes belonging to InitialGroup. The synchronization is modeled with the par operator (2). For instance, the beginning of Scene2 is associated with the end of Scenel (s_Scene2) (3 and 4). The same occurs with Scene2 and Scene3: the beginning of the latter is synchronized with the end of Scene2 (s_Scene3) (4 and 5). The disabling operator must also be mentioned (6). As one can observe, the req End event reaches all the processes of the group; it is used to model the end of the
282
application. When it is generated (by a transition to end), groups and scenes are successfully terminated (6).
process InitialG roupl[s_InitialG roup,e_l nitialGroup, lnteractlon,Data] (...,RI :StreamClass,dI :Time,d 2:Time, P1: BltmapClass, dPl:Time, P2:BitmapClass,dP2:lqme,P3:BibnapClass, dP3:Time,...):exit Is hide s Scene,s_Scene2,s_Scene3, req_End in s_InitialGroop; par s_Scene2# 2,s_Scene3# 2
3.3 Scene Representation Scene modeling differs in many aspects from group representation. One of the differences is that scene processes instantiate media objects and restrictions instead of groups and scenes. The presence of the loop operator in the representation is another important difference (1) (see figure 9). It is used to allow a scene to be presented more than once, which may happen when the application is logically organized as a net. Figure 9 shows Scene2, previously instantiated in figure 8. The req_Res event is responsible for restarting the media objects of the current scene when a transition to another scene occurs. The code that models a scene transition is composed of three events: s_Trans, req_Res and f_Scene (see figure 10a). The former denotes the occurrence of the transition. The second invokes the media objects of the scene to be reset. The third one indicates the end of the scene presentation. As the transition is an endless process, it is also disabled by the occurrence of the req_End event. When the transition is to the end of the application, the req_Res event is replaced by the req_End event (see figure 10b).
283
Fig. 9. Representation of Scene2 process Transibon [s_Trans,f_Scene,req_End,req_Res]:exit is loop forever in s Trans; req_Res;f_Scene endloop [> reel_End;exit endpror
process Transition [s_Trans,f Scene,req_End]:exit is s Trans,req_End;f_Scene;exit endpror
(a) Scene transition
(b) Transition to the end of the application
Fig. 10. Representation of transitions
3.4 Basic Objects and Temporal Restrictions Basic or monolithic objects were defined by [3] and model the presentation of simple media objects. These media objects are defined by the occurrence of synchronous (beginning, end) and asynchronous (user interaction) events. Several combinations of these events can be formulated, but only eight are pointed as important in the definition of interactive multimedia scenes. This work presents three of these combinations (see table 1). The fourth object presented in this table (pSsSe Synchronous start Synchronous end) does not appear in [3]. It allows time-dependent media with both minimum and maximum presentation durations to be modeled. In the definition of the processes, the Data event was used to represent the presentation of the media object.
284
c~-r
E-LOTO5 Code ~ ' i
II
l | ~1 I I
! E'IL~ ~. l l ~ ' d
I(
i I Klio
I L ' I ~ i l,
process pSsSestart,end, Data:class (media:class, d:time) :exit is start; Data(Imedia);wait(d);eod@tt=0;exit I I KIJ I i l l ! k l l
l;,l i I-'l:l'i
Used to model time-independent media objects like image and text with a known presentation duration,
I 141 l l f i l I I l l I I+'llitl~, iS( I I i I I I i i l a i
process pSm4mestart, eod,user, Data :class (media :class,d1,d2:time):exit is start; Data(!medla);wait(dl) ; {user@tt<=d2 wait(d2);exit);end@tt=0 l I I~1 i e l ! I+IL"I I ' ; I i I_+ 1L."lt'li I l l i I g l i i
Used to model user interaction; if the interaction does not occur dunng the interval dl,d2 the process is finished when the maximum time (d2) is reached. ;exit
I I+lii i i
process pSsAestart, end,user, Data :class (media:class,d:bme) : exit is start; Data(! medla);wait(d); user;eod@tt=O;exit
~roc
U I(lJ I I l l I k l l
II
Hodehng of user interaction without a maximum time to wait defined. The process finishes only when the interaction occurs.
I
I I ' ; l i i i i - ' I t ' J l l l l i I I l l i 1111I+'ll I I g ):41 i i I I I i i l i 4 1 i
process pSsSmestart, end,Oata :class (media:class,dl,d2:time) : exit is start; Data(! medla);walt(d 1);end@tL
Used to model time-dependent media objects like audio and video, which have a minimum and a maximum duration defined.
Table 1. Representation of basic objects Figure 11 shows the representation of P2, which appeared in the definition of Scene2 in figure 9. The event req_End (3) can again be observed, because media objects are also always being executed (1); if there is a loop in the scene definition, some media objects may be executed more than once during the presentation of the scene. In the same figure, one can also see the effect of the occurrence of the req_Res event: the restart of the media object to its initial state (2). process P2 s_P2,e._P2, Data, req_End, req_Res:exit is (P2:BitrnapClass,dP2:Time) loop forever in pSsSes_P2, e_P2, Data (P2,dP2) >req_Res
(1) (2)
endloop > req_End;exit
(3)
endproc
Fig. 11. Representation of the media object P2
The authoring model and consequently the tool, to be described in the next section, provide the definition of three distinct temporal restrictions: WaitMaster, WaitLatest and WaitEarliest. Their E-LOTOS representation controls the end of the media objects presentation that converge to the synchronization point. Restrictions are not implemented in libraries because their behaviour depends on the number of media objects that converges to the synchronization point. Figure 12 shows the representation of the WaitEarliest restriction.
285
process WaltEarliest[e_A,e_B,e C, e_Restriction,req_End,req Res]:exit is loop forever in
Fig. 12. Representation of the restriction WaitEarliest
4
The Authoring Environment
The creation environment is divided in two units: media repository and specification area. At any moment, the user can insert media objects into the repository. This is done by browsing a local media object or referencing a remote one. Figure 13 shows two windows that allow, respectively, the incorporation of new media objects (New Media) to the application and the manipulation (Medias Palette) of already existent ones.
Fig. 13. Management of the media objects of the application The specification area is composed of the scenes and groups of an application. Each scene is represented by both the temporal and the spatial views. The former allows the user to insert icons and synchronization points and to establish their relationship using arcs. The visible elements, used in the temporal synchronization, can be adjusted in the spatial view. Figure 14 shows the basic functionality of the authoring environment. The toolbar has shortcuts to its main functions (1). Two support windows can be observed: Specification Hierarchy (2) and Icons Palette (3). The former provides the user a general view of the application, presenting all the scenes and groups in a tree and providing a structured view of the relationships among them. In this case, the modeled application was the example presented in section 2 and is composed, therefore, of three scenes: Scenel, Scene2 and Scene3 (4). The latter, in its turn, provides mechanisms to visualize and edit the icons properties. In the same figure, the bitmap icon (P1) of Scene2 is selected and its specific properties are presented in the mentioned window. Icons that have an associated media object (audio, text, image, and video) present a special property called media. This property must be filled with a
286
media object existing in the repository. In this example, icon P1 is associated to the media object Rio Bonito.
Fig. 14. Interface of the authoring environment In figure 14, one can also observe the specification of Scene2 (5). It is composed of video (RI) followed by the sequential presentation of three images (P1, P2 and P3). By the end of the presentation of the last media object, a transition to Scene3 occurs. These information are presented by the temporal view. At the same time, the spatial view of Scene2 taking the icon P1 as reference is showed (6). It is possible to move or resize the visible media objects. Their properties related to coordinates and dimensions are automatically updated. Time-dependent media objects, like video, can be divided in smaller segments, allowing the synchronization of other elements with specific points of them. The environment provides mechanisms that make the process of fragmentation of these media objects easy. MUSE also provides means to reuse scenes and groups. It can be done by browsing the group or scene to be retrieved. It is necessary to redefine the transitions, defining where the application should evolve to after its presentation. Finally, it is also important to highlight the functionality of E-LOTOS code generation. This is obtained through the special saving option Save as E-LOTOS.
5
Conclusions and Future Work
This work initially proposed a new model for specifying interactive networked multimedia applications. Besides, mechanisms for mapping this model to the ELOTOS language were presented. Finally, the developed environment was described. The main contribution of this work is, therefore, the construction of an environment
287
turned to both ease of use and good expressiveness. At the same time, means to provide the formal representation of applications aiming at its analysis is also a great contribution. The model proposed distinguishes intentionally the concepts of logical structuring and temporal synchronization. The logical structure of the applications facilitates its organization in chapters, sections or in any other unit. For this reason, the application becomes modular, which contributes to lessen the complexity of the scenes and to avoid the occurrence of the state explosion problem. Future works include the creation of mechanisms that allow the user to define in the own environment parameters of quality of service, which will be used during the execution of the application. The possibility to define alternative media objects is also an important future task. It is important to highlight that the use of this environment integrated to the other tools under development in the project provides a complete framework, covering all the steps involved in the design of distributed multimedia applications: specification, verification and presentation. The ease of the authoring model and the use of a formal description technique to validate the applications turn the environment attractive and easy to use, without restricting the expressiveness of the environment.
References 1. G. Blakowski and R. Steinmetz. A Media Synchronization Survey: Reference Model, Specification, and Case Studies. IEEE Journal on Selected Areas in Communications, 14(1): 5-35, January 1996. 2. ISO/IEC DIS 13522-5. Information Technology - Coding of Multimedia and Hypermedia Information, Part 5: Support for Base-Level Interactive Applications, 1995. 3. N. Hirzalla, B. Falchuk and A. Karmouch. A Temporal Model for Interactive Multimedia Scenarios. IEEE Multimedia, 24-31, fall 1995. 4. L. Soares e R. Rodrigues. Autoria e Formataqao Estruturada de Documentos Hipermfdia com Restri~tes Temporais. In Proc. of the 3 rd Workshop on Multimedia and Hypermedia Systems, Sao Carlos, Brazil, May 1997. (In Portuguese) 5. P. Stnac, R. Willrich, P. de Saqui-Sannes. Hierarchical Time Stream Petri Nets: A Model for Hypermedia Systems. In Application and Theory of Petri Nets, 1995. 6. P. Stnac, M. Diaz and P. de Saqui-Sannes. Toward a formal specification of multimedia synchronization scenarios. Ann. Teltcommun. No. 49, pp 297-314. 7. ISO/IEC JTC1/SC21/WG7. Enhancements to LOTOS. Revised Working Drafts on Enhancements to LOTOS (V4), Project WI 1.21.20.2.3, January 1997. 8. J. P. Courtiat and R.C. de Oliveira. Proving Temporal Consistency in a New Multimedia Synchronization Model. ACM Multimedia, Boston, 1996.
Information Extraction and Database Techniques: A User-Oriented Approach to Querying the Web Zo@ Lacroix .1 and A r n a u d Sahuguet *.2 and R a m a n Chandrasekar***3 1 IRCS, University of Pennsylvania 2 CIS, University of Pennsylvania 3 IRCS & CASI, University of Pennsylvania
Abstract. We propose a novel approach to querying the Web with a system named AKIRA (Agentive Knowledge-based Information Retrieval Architecture) which combines advanced technologies from Information Retrieval and Extraction together with Database techniques. The former enable the system to access the explicit as well as the implicit structure of Web documents and organize them into a hierarchy of concepts and meta-concepts; the latter provide tools for data-manipulation. We propose a user-oriented approach: given the user's query, AKIRA extracts a target structure (structure expressed in the query) and uses standard retrieval techniques to access potentially relevant documents. The content of these documents is processed using extraction techniques (along with a flexible agentive structure) to filter for relevance and to extract from them implicit or explicit structure matching the target structure. The information garnered is used to populate a smart-cache (an object-oriented database) whose schema is inferred from the target structure. This smart-cache, whose schema is thus defined a posteriori, is populated and queried with an expression of PIQL, our query language. AKIRA integrates these complementary techniques to provide maximum flexibility to the user and offer transparent access to the content of Web documents.
Keywords: Web, data model, query language, information retrieval & extraction, agents, cache, view.
1 1.1
Introduction A user-oriented approach
T h e Web represents an immense reservoir of information, of great value if only we can m a n a g e it properly. This issue is a d a t a b a s e concern since it involves * Institute for Research in Cognitive Science, University of Pennsylvania, Suite 400A, 3401 Walnut Street, Philadelphia PA 19104, USA - Work supported by NSF STC grant SBR-8920230 and ARO grant DAAH04-95-I-0169. ** Department of Computer and Information Science, University of Pennsylvania, 200 South 33rd Street Philadelphia PA 19104, USA - Work supported by ARO grant DAAH04-95-I-0169 and ARPA grant N00014-94-1-1086. * * * Institute for Research in Cognitive Science & Center for the Advanced Study of India, Suite 400A, 3401 Walnut Street, Philadelphia PA 19104, USA.
290
representing and accessing information. The standard database approach focuses on the source(s) and provides a query language based on a given organization (schema) of the data: this is the source-driven approach. There have been various proposals to apply the source-driven approach to querying the Web. Some are oriented more towards the explicit structure of the Web (seen as a labeled graph), for example STRUDEL FFK+97 or WebOQL AM98; others, such as AltaVista, are based on a flat analysis (through keywords) of textual content. However, the Web does not behave like a standard database. There is no super-user in charge of monitoring the source(s) (the data is constantly updated), there is no homogeneous structure (and hence no common explicit structure), the Web itself never stops growing, etc. For these reasons, we believe that the sourcedriven standard approach is not suitable for the Web. As an alternative, we propose a user-oriented approach. When a user formulates a query in a given language, the only structure (of the source) used to answer the query is that which is explicitly expressed in the query. Therefore, instead of extracting in any way a "super schema" for the Web, why not simply use the schema explicitly expressed by the user? In AKIRA (Agentive Knowledge-based Information Retrieval Architecture), we propose exactly this: the system uses a target structure inferred from the user's query. 1.2
W h e r e and h o w to get t h e information
The Web provides information (Web documents) as well as information about this information (search engines, etc.). The AKIRA system uses them both. The relevant information needed to answer the user's query is gathered from one or more Web documents. Two problems immediately arise: (1) where to find the relevant documents? (2) how to understand their content? Useful services such as search engines, directories, etc. available on the Web help to solve the former, but return a lot of noise and redundancy. Their output must be filtered in order to select only relevant documents, a task related to problem (2). In AKIRA, we propose to process the content in order to locate the relevant documents and extract explicit or implicit structure from the retrieved documents. Our system has a flexible architecture based on agents to access all these services. 1.3
B e y o n d traditional information processing
The Web provides access to multimedia documents, but most of them rely partly or totally on textual content. Raw textual data is usually poorly analyzed through keywords: a "bag of words" representation of a document is often inadequate. Most Information Retrieval (IR) systems only index on words. A few systems use pattern matching or retrieval through phrases. However, these approaches do not capture any structure implicitly expressed in the text: no concept is extracted (for example, all names of persons may be indexed but no conceptual class Person is created), and no relationship (implicitly expressed by the order or position of words or sentences) is understood.
291
More advanced approaches could use syntactic or semantic tagging. Words could be annotated according to their part of speech categories (noun, verb, adjective, etc.) or grouped into categories such as Person to create concept classes. For instance, in the sentence "Barbara Pernici is the Program Chair or CAISE98", Barbara Pernici and Program Chair are respectively instances of concept classes Person and Designation. The next step would be to provide some relations between these concepts by introducing meta-concepts. Techniques from computational linguistics may be used for this purpose. In our example, classes Person and D e s i g n a t i o n are connected through the relation Appointment. SuperTagging JS94, which provides rich syntactic labels, may be used with other tools (for example a co-reference tool BDR+97) to capture a variety of syntactic and semantic information, and thus meta-concepts such as Appointment.
1.4
Using D B techniques
The user-oriented paradigm means that the structure through which the data is viewed does not come from the source but is extracted from the user query. When a query is evaluated, the relevant documents are retrieved from the Web and stored as is. Then the information is extracted from the raw data using computational linguistic techniques. The AKIllA cache (smart-cache) stores these extracted layers of meta-information on top of the raw data. The smart-cache is an object-oriented database whose schema is inferred from the user's target structure. It is designed on demand out of a pool of conceptual schema components that can be assembled together to match concepts and meta-concepts required in the user's query. The smart cache can be seen as a view of the Web. The use of a database as a cache provides the system with a real query language PIQL (an extension of OQL Ba97 with path expressions ~ la POQL CCM96) to query the Web. The paper is organized as follows. Section 2 presents an overview of currently available Web representations and query languages, and sketches an outline of AKIRA. Section 3 presents a motivating example, defines the central concept of Web views and shows how AKIRA models some of the components required to evaluate a query. The architecture of the AKIRA system is described in Section 4. The last section contains our conclusions. 2
Web
Query
Languages
We believe that a high-level Web query language should be provided to the user. This should allow the user to specify exploration plans to generate automatic browsing and analysis of the content. Successive steps of the navigational exploration should be performed transparently by the system on behalf of the user. In this section, we first analyze the limits of available languages, investigate how the content of Web documents should be accessed by the queries and propose our representation of the Web and our query language.
292
2.1
W e b representations and languages
The expressive power of a query language reflects the expressiveness of the underlying representation of information. The Web consists of documents (text, picture, audio, video, etc.) with a HTTP address (a URL). Seen as a table URL, content, the only language available to query the Web is the "type-fetch'n read" language (see Table 1). The user actually types a query (which is nothing but a URL) and gets back one or more documents, or nothing. Within this representation, neither the meaning of the URL nor its content is understood or accessed by this primitive language. The user has to provide both the reading and understanding. The "type-fetch'n read" language is nothing but browsing. Representation (data type, model and organization) rel table URL, content rel table URL, content and table URL, label, URL rel !table URL, title, text, type, length, modif hypergraph I and table URL, label, URL O 0 object=page attribute=label O 0 object=page attribute=HTML (or source-driven structure) hypermediaooobjects=instances of concepts attribute=meta-concepts
Language "type-fetch 'n read" FO, Datalog AV97 W3QL KS95 WebSQL MMM97 WebOQL AM98 Penelope AMM97 PIQL
Table 1. Web Representations and Languages
Web documents can also be viewed as a set of graphs not (fully) connected in any regular manner. Thus there is no a priori way to search for something. There is no expectation that pages will be connected to any other page, nor that any page will have links outside the page. The only way to explore is to start from a given point (a valid URL, a bookmark, etc.) and navigate from page to page, accumulating addresses of new pages along with knowledge about the Web itself. The usual way to go through this hypergraph is browsing (choosing a hyperlink to jump to, by reading the content of the page) and repeating this process until satisfactory answers are reached. This process is non-deterministic since it is based on successive choices of hyperlinks that may lead to the expected information. More advanced technologies may enable automatic (and thus deterministic) browsing. Based on a representation matching the hypergraph (labels) and some additional structure extracted from the source (HTML tags in particular), the proposed query languages express browsing queries based on a thorough knowledge of the hypergraph (where the user must know a priori the labels!) AV97,KS95,AM98 and the extracted structure MMM97,AMM97. These languages can express queries such as "retrieve all pages accessible from the
293
URL http://www.pianosa.cnuce.cnr.it/ with a label path expression ending with caise98" (a label path expression is a list of labels). These languages are limited because their representation of the Web is not compatible with a high-level Web query language that would access the content of Web multimedia documents. Instead, the user must cope with reading everything to extract relevant information and discarding anything spurious. 2.2
Accessing the content
What kind of information is available on the Web? One can find any information, from unstructured documents to highly structured ones (databases), from raw information to information about the Web itself (search engines). But none of the proposed languages associated with the representation of the Web as a hypergraph (see Table 1) really automatically accesses or understands all this information. Various tools have been successfully developed to index specific media formats. Search engines such as Yahoo, AItaVista, Excite, Infoseek or HotBot for text, WebSeer or tycos for images, give access to pre-computed indexes of Web sources. But is querying a search engine querying the Web? A user may use a search engine to query the Web; in this case, no knowledge of the hypergraph is required. A query to a search engine is nothing but a parameterized URL. For instance, looking in Yahoo's base for "CAiSE" corresponds to typing the following URL: h t t p : / / s e a r c h . y a h o o . com/bin/search?p=CAiSE. We regard search engine querying as parameterized browsing through databases which are preprocessed and materialized views of the Web. These databases only represent a view of a portion of the Web. Only documents visited by spiders and indexed are returned by queries to search engines. Their visibility comes out by being connected to other visible parts of the Web, or by intentionally being registered. Moreover, some specific (types of) sites are not indexed on purpose. It follows that querying a search engine is limited because (1) it restricts the query to data stored in a given database, and (2) the structure of the database only captures shallow information about the structure of the content of the document (via its indexes). AKIRA takes advantage of usual search engines as well as new technologies from computational linguistics. 2.3
A Smart-Cache
We focus on the idea of querying the Web on-the-fly, which means that the Web is not preprocessed, nor is the query restricted to any stored subset of the Web. This is in contrast to ACC+97,AMM97, where parts of the Web are preprocessed and stored in a database, for which the schema is defined according to its sources. The resulting database is no longer linked to its Web sources (that may be updated meanwhile). Thus, a query in their system is evaluated against a closed-world database, and not against the Web. A cache usually stores HTML pages in a dummy file-system where manipulation is based on page names. AKIRA replaces the standard browser cache by
294
a smart-cache, which is an object-oriented database. Its smart cache is empty and unstructured before any query. The schema is inferred from the user's query (target structure). Classes are populated with information extracted from documents retrieved from the Web. Most of the above representations (see Table 1) are page oriented to match the Web represented as an hypergraph. However, in order to take advantage of Information Extraction (IE) techniques to analyze the content of Web information and extract information with respect to a given structure, a finer granularity is required. We choose a flat representation (as opposed to a tree-like representation) based on fragments (see Section 3.6). A fragment represents an indexed part of a multimedia Web document. A concept is represented as an abstract object class populated with instances referring to several co-referent fragments. Metaconcepts are non-valued attributes between concept classes. Our smart-cache is a Web view. We adopt PIQL (Path Identity Query Language), an enrichment of the standard object query language OQL Ba97 with identity invention ~ la IQL AK89 and with generalized path-expressions ~ la POQL CCM96. Our model of Web view with its query language provides a rich unified formalization of the Web, useful for future optimization. We motivate our work with a detailed example in the next section. 3
AKIRA's
Web Views
In this section, we illustrate AKIRA's approach with a motivating example. We assume the reader is familiar with the concepts of object-oriented databases as presented in AHV95. In AKIRA, a Web view is an object-oriented database instance. The object schema consists of classes, a class hierarchy (subclassing relationship), attributes and methods with their signatures. Classes are populated with object identifiers (oids). AKIRA's view mechanism is inspired by the view mechanism introduced in dSDA94,LDB97. 3.1
Motivating
example
Researchers are usually interested in Calls for Papers (CFPs) in subject areas related to their research. While CFPs are widely disseminated, being published in journals, newsgroups, mailing-lists etc., researchers would appreciate a facility to find out about relevant CFPs at any given time. Calls for Papers are usually text documents with some minimal implicit structure. Typically, they provide information about: - the - the - the - the
name, venue and dates of the conference/seminar/workshop, list of topics of interest, contact people to whom submissions are to be made, last date for submissions, etc.
The AKIRA approach may be used to identify conferences of interest to particular users, using a query such as expressed in Example 1.
295 Poolof schema Smart-Cache components I i I I ! I
I user'squery~-~ PIQLqueryI
>
i
queryresult
,
! . . . . .
The WEB
..
"'1Retrieval i ~
J
ExtractionI
Fig. 1. AKIRA's query processing.
Example 1. The user wants information about upcoming conferences such as query Q I : "Conferences about Information Systems with a submission deadline after July 31, 19987" As illustrated in Figure 1, we first extract the concepts expressed by the user in his query (see Section 3.3 and Section 3.4). We describe how the relevant documents may be identified in Section 3.5 and how they are processed as explained in Section 3.6 to populate the concept classes in Section 3.7. 3.2
P o o l o f schema components
Before further describing the evaluation of a query, it is important to emphasize the capabilities of our approach. When querying a database, a user is restricted to a given and frozen organization of information, enforced by the creator when designing the schema. Should the user send a request beyond the schema, he will be denied access to the expected information. The creator has imposed his view to the user. This is the limitation of the source-driven approach. Our user-oriented paradigm grants more flexibility by allowing the user to design his own view on demand. There is no magic: the limits are transfered to the extraction capabilities (see Section 3.6) of the system. AKIRA's administrator is in charge of representing these capabilities at a conceptual level in terms of schema components, in a modular way. He provides the user with a pool of schema components that can be combined to specify the user's view. An IE tool capable of identifying conference names is represented by a concept class C o n f e r e n c e with attribute name. Similarly, a date extraction tool corresponds to a class Date with attributes month, day and y e a r . Each of these classes is a schema component by itself. A concept class can be specialized according
296
to other extraction capabilities. For example, attribute t o p i c , listing topics of interest, can specialize class Conference. Two concept classes can be also combined through a meta-concept such as s u b m i s s i o n _ d e a d l i n e to assemble a new conceptual schema as illustrated in Figure 2.
Conference
submissiondeadline
I name topic
Date
t
~day~ year month
Fig. 2. Conceptual schema.
3.3
Query processing
AKIRA's framework is compatible with a Natural Language (NL) user interface such as described in ART95. In pattern-matching systems, a relational table is assumed and natural language patterns are associated with action rules. Similarly, in AKIRA, action rules can correspond to concept classes, as illustrated below. pattern: action:
... "conference name" ... select c.name from c in Conference
Action rules built for each schema component permit us to process query Q1 and obtain a corresponding PIQL expression: select c.name from where
c in Conference "Information Systems" in c.topic and c . s u b m i s s i o n _ d e a d l i n e . m o n t h > 7 and c.submission_deadline.year
= 1998
The NL interface matches the words expressed in the NL query with the user's target structure (see Figure 2). A standard NL database interface does not require the user to know the organization (schema) of data. Therefore AKIRA also provides action rules that translate patterns into generalized path expression k la P O Q L CCM96. Suppose that a user wants to know "Conferences where the temperature is over 9OF'. There is no built-in attribute t e m p e r a t u r e available for class C o n f e r e n c e in the pool of schema components, however the system will translate the query using the pattern: . . . " c o n f e r e n c e name" . . . temperature associated with the action: select c.name from c in Conference where c.*.temperature>90
297
where c. *. temperature>90 is a general path expression. If attribute c o u n t r y is available at Conference and attribute temperature at class Country, then the system will infer from: s e l e c t c.name from c in Conference where c. *
.temperature>90
the OQL expression: select c.name from c in Conference where c.country.temperature>90
3.4
View mechanism
Our view mechanism goes through a pipeline (see Figure 3) of successive and interleaved views (obtained by successive materialized extensions dSDA94,LDB97). The main task consists in specifying the schema transformation from the current schema to the target structure. When the first query is asked, the current schema is empty. In case of a refinement, a current schema (defined to answer previous queries) structures the cache and has to be extended to support the target structure (by adding new classes and/or attributes). The input of the view mechanism is a PIQL query together with the current schema of the cache (if any). First the target structure has to be inferred from the PIQL query. In particular, the system has to resolve the general path expression (if any) by introspecting its pool of schema components for all possible matching paths. The view specification is derived as the difference between the target structure and the current schema. The view mechanism forwards three queries to structure and populate the cache: 1. a query invoking IR tools to retrieve relevant documents from the Web, 2. a schema transformation query defining the new structure of the cache (new classes and/or attributes) according to the user's target structure, and 3. an update query triggering methods that invoke IE tools to populate the cache using the content of retrieved documents. 3.5
R e t r i e v i n g Relevant Information
To answer Q1, we need to populate the cache, namely to identify pertinent CFP information through the following steps. Information Retrieval: we can look for documents indexed by search engines which satisfy a query expressed as a boolean expression on keywords/phrases such as: "Calls for Papers" OR "Call for Papers". W e can also use websites and newsgroups which collateinformation about conferences in one or more subject areas. For example, one can find a listof C F P s about W W W , Hypertext, Structured Documents, Information Management~ Authoring/Reading Tools, Reuse of W e b Information, metadata, etc. at the U R L
298
http ://www. mel. dit. csiro, au: 8080/,,~delloro/db/. These (typically volunteer) efforts are quite useful, but not always up to date, and not expected to be exhaustive in any way. For a variety of reasons, the A K I R A approach is likely to be significantly better than using such repositories per se.
Information Filtering: in a second step, we discard documents which are likely to be spurious or lacking in relevant information. These include documents which do not contain the standard CFP phrases, documents which are Web-redirection documents, empty documents etc. We may also discard documents which contain the word Archive (these may be mega-Archives without much relevant content). A filtering tool such as GLEAN C897 may be used for this purpose.
3.6
Extracting Information: fragments
From retrieved documents, we identify names of meeting and dates thanks to our IE agents. A conference is identified by its name and has a canonical representation expressed by an acronym (for example CAISE98). A date is a string of characters expressing a month, a day and a year. Its canonical representation (aka normalized representation) is a list of three integers (for example 11,30,1997). We introduce the notion of fragment, which is a flexible way to consider a document in different granularities, according to the needs of the user. Fragments correspond to the strings of characters indexed by IE agents in the retrieved documents as illustrated in Figure 3. Each fragment is characterized by a pair consisting of a document name and a span (a span consists in turn of a pair of integers specifying the starting and ending position of the indexed string of characters in the document Gri96). When the fragmentation is accomplished, concept classes may be populated.
original
new instances of class Conference
document
mr->
"Conference"
Agent
Fig. 3. AKIRA's fragmentation pipeline.
new instances of class Date
299
3.7
Concept classes
As explained in Section 3.4, the target structure inferred from query Q1 specifies the object schema of the smart-cache as follows. Class Conference
Each extracted conference name is represented by its canonical form. For instance, fragments such as CAISE, lOth Conference on Advanced Information Systems Engineering, etc., are represented as an object instance of class C o n f e r e n c e . The value of its attribute name is its canonical representation CAISE98, and the value of its attribute f r a g m e n t s , the set of all fragments it refers to. Class Date is similarly populated. The value of extra attributes such as t o p i c is extracted by an IE tool (for example, a zoner a that extract zones mentioned as "Conference Topics", etc.) from CFPs. For each instance of a conference, the value of attribute t o p i c is the set of all topics extracted from the "Conference Topics" zone of its CFP. Meta-concepts such as s u b m i s s i o n _ d e a d l i n e also invoke IE tools to extract the relationship between two concepts. For example, from the CFP of a conference, a zone "Important Dates" can be identified from which the submission deadline can be extracted. Another tool may exploit Super Tagging JS94. A training phase consists in extracting patterns from the sentences where the submission deadline is expressed (such as "All submissions must be sent to the PC chair by November 11, 1997" or "Authors are invited to submit a position paper no later than November 11, 1997", etc.) in a sample of CFPs. The extraction phase consists in (1) retrieving sentences where "send", "submit", etc. occur (with a grep) and comparing their pattern with the ones obtained from the training session; and (2) extracting the date from each sentence that matches a pattern and identifying the conference submission deadline.
4
AKIRA Architecture
The AKIRA system can be viewed as a personal proxy that provides the user with transparent access to the Web: the input to AKIRA is provided through a 1 See AI97 for zoning extraction tools.
300
standard HTML form or through a parameterized URL while the output is an HTML page generated on-the-fly by the system. These "virtual" pages, similar to the virtual documents in [VDH97], can be bookmarked and reconstructed on-demand. The AKIRA system basically receives a query, creates an object-oriented database (a Web view), and returns the output of the query against the instance of the database. It has five components: the D i s p a t c h e r , the D B M S (DataBase Management System), the V i e w F a c t o r y , the A g e n t Pool, and the O u t p u t Formatter as illustrated in Figure 4.
Fig. 4. AKIRA's architecture. The Dispatcher has a role similar to the one of a query processor for a database management system. It translates the user's query in a PIQL expression and extracts the target structure. The View F a c t o r y is an essential part of the system. The View Factory's task is to populate the cache with information extracted from documents retrieved from the Web by IR agents. The D a t a b a s e S y s t e m (DBMS) storing the expected Web view is objectoriented. It is defined with a view expression sent by the View Factory which specifies its schema as well as its population. The A g e n t P o o l contains IR, IE, formatter agents, etc. IR agents consist of wrappers to correspond with data sources available on the Web (search engines or services), and information filtering tools such as GLEAN [CS97]. IE agents extract concepts and meta-concepts. IE agents such as conference acronym and location recognizers together with a co-reference tool identify concept instances. SuperTagging [JS94], which provides rich syntactic labels, and zoners extract
301
meta-concepts. Formatter agents can be of type summarizer, table-of-content, glossary, etc. The O u t p u t F o r m a t t e r , is used to format the output according to the user's needs. The motivating CFP example provides only a glimpse of the range of capabilities of the AKIRA system. 5
Conclusion
In this paper, we have described AKIRA, an alternative approach to querying the Web. Here are some of the several benefits to using the AKIRA framework: 1. Benefits from Natural Language techniques: Techniques from natural language processing provide access to explicit as well as implicit structure of textual content. Some of the ideas we are discussing have been proposed in other contexts (for example, ISLE97). 2. Benefits from Database techniques: The separation between the logical view (concept and meta-concepts) of Web documents and its storage in the smart-cache presents several advantages, including a Web query language. Its schema is tailored by the user when asking a query. Our approach does not require the integration of several heterogeneous sources in a global common representation. Moreover, it is worth noting that AKIRA does not assume that it can start from a database representation (schema and instances) of the Web like many other systems dealing with site-restructuring (see for instance FFLS97,AM98,GW97). 3. Benefits from the AKIRA architecture: AKIRA offers a transparent architecture to access data of various media from the most loosely structured sources (newswire, press release, personal homepages or newsgroups) to highly structured sources (legacy databases, catalogs, digital libraries). Its modular framework and extensible design provides the user with a highly tunable interface to the Web. We present two important directions for future work. U n d e r s t a n d i n g h y p e r m e d i a d o c u m e n t s : Web documents are multimedia and our conceptual representation is medium-independent. AKIRA will take advantage of various tools successfully developed to index specific media formats. IE tools usually parse linear textual documents. They should first be generalized to mark-up language syntax (SGML, HTML, XML, etc.) in order to understand and use the meta-organization provided by tags. Moreover, a Web document is no longer a single linear page but a hyperdocument (a graph of connected nodes). IE tools should be able to extract structure from a hyperdocument and thus over hyperlinks. AKIRA's approach aims at automating browsing. When IE tools can adjust the hyperstructure of Web documents, heuristics can be introduced to select hyperlinks according to a strategy which may be used to mimic human browsing.
302
AKIRA can take advantage of knowledge representation. For instance, by using a topic hierarchy and a thesaurus, AKIRA can be programmed to retrieve information about particular subject areas and all its super-areas. An approach combining knowledge representation and natural language processing such as conceptual indexing Woo97 could dramatically improve AKIRA's ability to retrieve relevant information. Q u a l i t y of service: AKIRA's system is subject to the inherent hazards of information processing techniques (recall/precision). However, it aims at delivering information together with a measure o,f confidence. Our deliberate choice of processing data on-the-fly forces us to emphasize the issue of performance. Standard database query rewriting can be considered to optimize the evaluation of the query on the database instance CCM96. The view mechanism itself may be tuned according to both the view definition and the retrieval of documents. Other approaches to manage semi-structured data such as Lorel AQM+97 could be investigated. The AKIRA system LSC98 is under development at the Institute for Research in Cognitive Science in collaboration with the Database group of the University of Pennsylvania.
Acknowledgment: Alberto Mendelzon and Anne-Marie Vercoustre are thanked for valuable comments on an earlier version of the paper.
References ACC+97
S. Abiteboul, S. Cluet, V. Christophides, T. Milo, G. Moerkotte, and J. Simeon. Querying documents in object databases. Journal on Digital Libraries, 1997. AHV95 S. Abiteboul, R. Hull, and V. Vianu. Foundations of Databases. AddisonWesley, 1995. AI97 D.E. Appelt and D. Israel. Building information extraction systems. In ANLP-97 Tutorial, Washingtoa, D.C., March 1997. AK89 S. Abiteboul and P. Kanellakis. Object Identity As A Query Language Primitive. In ACM SIGMOD Symposium on the management of Data, pages 159-173, Portland Oregon USA, June 1989. AM98 G. Arocena and A. Mendelzon. WebOQL: Restructuring Documents, Databases and Webs. In Proceedings of the International Conference on Data Engineering, Orlando, February 1998. AMM97 P. Atzeni, G. Mecca, and P. Merialdo. To Weave the Web. In Proc. of Intl. Conf. on Very Large Data Bases, Athens, Greece, August 1997. AQM+97 S. Abiteboul, D. Quass, J. McHugh, J. Widom, and J.L. Wiener. The Lorel Query Language for Semistructured Data. Journal on Digital Libraries, 1997. ftp://db.stanford.edu/pub/papers/lore196.ps. ART95 I. Androutsopoulos, G.D. Ritchie, and P. Thanisch. Natural language interfaces to databases - an introduction. Journal of Natural Language Engineering, 1(1):29-81, 1995. Cambridge University Press. http://www.mri.mq.edu.au/ion/nldbsurv.ps.gz.
303
ART97
AV97 Ba97 BDR+97
0CM96
cs97 dSDA94
FFK+97
FFLS97
Gri96
GW97
JS94
KS95 LDB97
LSC98
MMM97 RC93
I. Androutsopoulos, G.D. Ritchie, and P. Thanisch. A framework for natural language interfaces to temporal databases. In Proceedings of the 20th Australasian Computer Science Conference, volume 19(1), pages 307-315, Sydney, Australia, 1997. Australian Computer Science Communications. http://www.mri.mq.edu.au/ion/acsc97.ps.gz. S. Abiteboul and V. Vianu. Regular Path Queries with Constraints. In Proc. A C M Syrup. on Principles of Database Systems, 1997. D. Bartels and al. The Object Database Standard: ODMG 2.0. Morgan Kanfmann, San Francisco, 1997. B. Baldwin, C. Doran, J.C. Reynar, B. Srinivas, M. Niv, and M. Wasson. EAGLE: An Extensible Architecture for General Linguistic Engineering. In In Proceedings of RIAO'97, Montreal, June 1997. V. Christophides, S. Cluet, and G. Moerkotte. Evaluating Queries with Generalized Path Expressions. In Proc. ACM SIGMOD Syrup. on the Management of Data, 1996. R. Chandrasekar and B. Srinivas. Using Syntactic Information in Document Filtering: A Comparative Study of Part-of-speech Tagging and Supertagging. In In Proceedings of RIAO'97, Montreal, June 1997. C. Souza dos Santos, C. Delobel, and S. Abiteboul. Virtual Schemas and Bases. In Proceedings of the International Conference on Extending Database Technology, March 1994. M. Fernandez, D. Florescu, J. Kang, A. Levy, and D. Suciu. STRUDEL: A Web-site Management System. In ACM SIGMOD - Research prototype demonstration, Tucson, Arizona, May 1997. M. Fernandez, D. Florescu, A. Levy, and D. Suciu. A Query Language and Processor for a Web-Site Management System. In A CM SIGMOD Workshop on Management of Semistructured Data, Tucson, Arizona, May 1997. R. Grishman. TIPSTER Text Phase II Architecture Design. Technical report, TIPSTER Text Program, 1996. http://www.tipster.org/docs/arch23.ps.gz. R. Goldman and J. Widom. DataGuides: Enabling Query Formulation and Optimization in Semistructured Databases. In Proc. of Intl. Conf. on Very Large Data Bases, Delphi, Greece, August 1997. to appear. A.K. Joshi and B. Srinivas. Disambiguation of Super Parts of Speech (or Supertags): Almost Parsing. In Proceedings of the 17~h International Conference on Computational Linguistics (COLING '9~), Kyoto, Japan, August 1994. D. Konopnicki and O. Shmueli. W3QL; A query system for the World Wide Web. In Proc. of Intl. Conf. on Very Large Data Bases, 1995. Z. Lacroix, C. Delobel, and Ph. Br~che. Object Views and Database Restructuring. In Proc. of Intl. Workshop on Database Programming Languages, August 1997. Z. Lacroix, A. Sahuguet, and R. Chandrasekar. User-oriented smart-cache for the Web: What You Seek is What You Get! In ACM SIGMOD Research prototype demonstration, Seattle, Washington, USA, June 1998. http://www.cis.upenn.edu/,-,AKIRA. A. Mendelzon, G. Mihaila, and T. Milo. Querying the World Wide Web. Journal on Digital Libraries, 1(1):54-67, 1997. S. Ramani and R. Chandrasekar. Glean: a tool for Automated Information Acquisition and Maintenance. Technical report, NCST Bombay, 1993.
304
SLE97
VDH97 Woo97
J. Shakes, M. Langheinrich, and O. Etzioni. Dynamic reference sifting: A case study in the homepage domain. In Proceedings of the Sixth International World Wide Web Conference, pp.189-200, 1997), 1997. A-M. Vercoustre, J. Dell'Oro, and B. Hills. Reuse of Information through virtual documents. In Proceedings of the 2 nd Australian Document Computing Symposium, Melbourne, Australia, April 1997. W.A. Woods. Conceptual indexing: A better way to organize knowledge. Technical Report TR-97-61, Sun Microsystems Laboratories, April 1997.
Goal-Driven Business Process Analysis Application in Electricity Deregulation V. Kavakli and P. Loucopoulos
Department of Computation U.M.I.S.T. PO Box 88, M60 1QD, Manchester, UK {kavakli I pl} @co.umist.ac.uk
Abstract
Current business challenges such as deregulation, mergers, globalisation and increased competition have given rise to a new process-centric philosophy of business management. The key issue in this paradigm is the concept of business process. From a methodological perspective, this movement has resulted in a considerable number of approaches that encourage the modelling of business processes as a key component of any improvement or reengineering endeavour. However, there is a considerable controversy amongst all these competing approaches about the most appropriate way for identifying the types and number of relevant processes. Existing business process modelling approaches describe an enterprise in terms of activities and tasks without offering sufficient guidance towards a process-centred description of the organisation. In this paper we advocate the use of a goal-driven approach to business process modelling. A systematic approach to developing and documenting business processes on the basis of the explicit or implicit business objectives is put forward. We argue that such an approach should lead to a closer alignment between the intentional and operational aspects of an organisation. Our approach is exemplified through the use of parts of a large industrial application that is currently making use of a goal-driven business process modelling.
1
Introduction
The traditional practice o f managing an enterprise adopts a functional view in which the business is organised along individual types o f work performed, resulting in organisational structures which reflect the particular functional view adopted by the business. The main reason for adopting a functional organisation is the achievement o f maximum performance o f individuals or business functions. Nevertheless, this inward focus on 'internal' performance rather than 'global' efficiency suffers from a number o f drawbacks, especially when business improvement is sought. In particular, improvements occur piecemeal and independently o f one another, while concentration on the symptoms o f one function ignores causes in important crossfunctional interdependencies.
306
Current business challenges such as deregulation, mergers, globalisation and increased competition, have given rise to a new philosophy of business management that organises an enterprise in terms of processes rather than functions and tasks. The basic characteristic of this approach is the re-orientation of business from performing as a cluster of functions or divisions to integrating activities within a limited number of core processes. Each core process captures cross-functional interdependencies and concentrates on few strategic objectives that determine competitive success. Therefore, a process centred approach links improvement efforts in different functions to a shared set of strategic objectives. Adopting a process view however, requires suitable tools for identifying, modelling and measuring business processes. Existing business modelling approaches describe enterprises in terms of activities and tasks offering little or no guidance towards a process-centred description of the organisation. In this paper we advocate the use of a goal-driven approach whereby a business is seen as a purposeful system aiming to achieve defined objectives which add value to its customers. This approach is part of a larger enterprise knowledge modelling framework, known as the EKD approach Loucopoulos, Kavakli, et al 1997. Allied to business process modelling is the larger issue of business change itself. Business change is also seen as goal-driven in EKD; the need for business change is externalised in terms of strategic business goals, which in turn shape business processes. Therefore, business change management is the process of identifying the business goals for change and analysing the impact that these goals have to business processes. The paper is organised as follows. Section 2 introduces the industrial application which is referred to throughout the paper. Section 3 introduces the notion of business process in terms of its defining characteristics and presents a critique of existing process modelling techniques. Section 4 briefly introduces the goal-driven approach to business process modelling. The application of the approach is illustrated in section 5, using examples from the industrial application introduced in section 2. Finally, section 6 concludes with a discussion on the role of goal-driven business process modelling within the broader context of business change management.
2
Background to the Application
The work presented in this paper is part of a big industrial application that concerns de-regulation of a large European electricity company. The company is divided in three operational areas generation, transmission and distribution. Generation is responsible for the production of electrical power. Transmission is responsible for the high voltage transport of electricity. Finally, distribution is responsible for the medium voltage (M/V) and low voltage (L/V)
307
transport of electricity, its delivery to consumers and the merchandising of electricity services. These areas operate under the rules and regulations of a governmental regulatory body that controls issues like tariffs, production levels, environmental policies, etc. Currently the company operates in a total monopoly market which means that it is the single operator of all three areas. A high-level view of the main company actors and their roles is illustrated in Fig. 1. Generation Operator r
I Customer
I
Transmission Operator
~ "~ Electricity Generation Produce
c
Distributor
R Supply l electricity l .
electrical
power
Buying Electricity
IRegatorI r
Buy electricity
9
Regulation Regulate
electricity market
I
Fig. 1. Main company actors and their roles in the monopoly market In anticipation of the opening of the European electricity market, the company is in the process of re-designing its business structure and planning reforms for the future, in order to increase its competitiveness and retain its market share. This is especially critical in the distribution area which is the interface of the company with the final customer. Adopting a process view of the business is a key factor in this effort. Experience from previous projects in the company has shown the need for a structured approach for describing and measuring the business processes. Nevertheless current methods focus on what it is done (the tasks and activities performed) rather than how work is done in terms of processes, offering little assistance in this direction. This study reports on the application of a goal-driven approach whereby business goals are put forward while identification and analysis of business processes is based on their intentional affinity. For the purpose of this paper we focus on one section of the distribution area, namely the Distribution District. The current structure of a Distribution District is organised along four distinct functional sections illustrated in Fig. 2: the Technical Section, the Customer
308
Electrification Section the Personnel Section and the Customer Services Section (or agencies). District
I
I
Technical Section
Customer Electrification Section
I
I
I
Personnel Section
Customer Services Section
Fig. 2. Functional organisation of a District
The Personnel Section deals with internal matters of District employees, including safety and training issues. The Customer electrification section mainly plays a manager role. It is responsible for checking and checking all expenditures and authorising the construction of works that concern the electrification of customers as well as the managing of customer payments to the company. The executive roles are played by the Technical Section. The Technical Section is responsible for the operation and maintenance of the distribution network, as well as the technical servicing and maintenance of customer installations. Finally the Customer Services Section plays mainly administrative roles being the interface between the electricity consumer and the District. In addition the customer services section performs periodical readings of the electricity metering devices at customer installations in order to calculate electricity consumption and receives customer payments.
3
Business
Process Modelling
The concept of business process is a key issue in the process centred paradigm. However, there is a considerable controversy around the number and types of processes appropriate to a given organisation Davenport 1993. The difficulty derives from the fact that there exists no explicit way for determining business processes. There is a lack of a coherent and universally accepted definition of what a business process actually is. Nevertheless, there are some common features of business process definition in the literature Alderman, Maffm, et al 1997; Davenport 1993; Hammer and Champy 1993; Ould 1995 that provide guidance as to how business processes should be defined. In summary a business process in the process-centred organisation demonstrates the following characteristics: a business process has well identified products and customers, such that business objectives are matched through the (product offering) business process and delivered in the form of the product; customers may be external or internal to the organisation; products may include finished goods or services
309
9 9 9
a business process has goals, i.e., it is intended to achieve defined business objectives aiming to create value to customers a business process involves several activities which collectively achieve defined business process goals and create value to customers a business process crosses functional/organisational boundaries; it concerns the collaboration between organisational actors that are contributing to (or constraining) the satisfycing of business objectives
In these terms a business process constitutes the manifestation of what organisational actors do in order to achieve business objectives. Organisational actors include individuals or groups which may be internal or external to the organisation (e.g., company employees, organisational departments, customers, suppliers etc.) and influence the realisation of business objectives. Business objectives aim at creating value to customers in other words they concern customer value goals. Business process modelling is a generic name that refers to a collection of techniques which are used to model the behaviour of business systems. Existing process modelling approaches mainly originate from the software engineering field and fall in one of three categories: 9
9
9
Activity-oriented approaches describe a process as a set of ordered activities
(e.g., SADT Ross and Schoman 1977, IDEF0 IDEF0 1993, DFDs DeMarco 1978, Workflows Swenson and Irwin 1995, the F3 process model Bubenko 1994). The emphasis is on what activities take place. Each of these activities is decomposed in smaller tasks corresponding to smaller steps in the process. In addition to a collection of tasks activity-oriented models define the order of task invocation or condition(s) under which tasks must be invoked, task synchronisation, and information flow. Agent-oriented (or role-oriented) approaches specify and analyse the role of the agents that participate in the process (e.g., Role Interaction Nets Singh and Rein 1992, Role Activity Diagrams Ould 1995, the i* model Yu 1994, the ORDIT approach Dobson, Blyth, et al 1994). The focus is on the entity that performs a process element. Roles represent the sequences of activities carried out by agents engaged in a co-operative behaviour. Product-oriented approaches represent a process through the evolution of its products (e.g., Easterbrook and Nuseibeh 1995, Franckson and Peugeot 1991). Product oriented models do not put forward the activities involved in a process but rather the result of these activities. The focus is on products and transformations made on them. Each product entity has a defined sequence of states and triggers that cause state transformations.
All the above approaches promote a view of a process that is based on the notion of activity. Activity-oriented approaches focus solely on description of activities. In addition product-oriented approaches couple activities to their output (the product),
310
while agent-oriented approaches establish an explicit link between the activities and the agent responsible for these activities. Existing approaches offer little guidance for identifying business processes. In activity-oriented approaches the main mechanism for grouping activities into processes is that of composition/de-composition. This mechanism however, does not offer a unique way to identify a process. The difficulty derives from the fact that processes are almost indefinitely divisible; the activities involved in fulfilling a customer order, for example, can be viewed as one process or hundreds. Agentoriented approaches on the other hand, group activities into processes according to the organisational agent that performs these activities. Yet, a process may cut across the organisation involving several organisational agents. Finally, product-oriented approaches group activities based on the product that they manipulate and this notion of a process is in accordance with the suggested business process definition as the delivering of products to customers. However this focus on product rather than organisational behaviour fails to describe other important components of a business process such as the business goals that the process intends to achieve and the collaboration of the agents that contribute to the realisation of process goals.
4 4.1
The EKD Approach to Business Process Modelling Overview
It becomes obvious that taking a single modelling perspective (product, activity or role) is not sufficient for expressing business processes. A different approach towards business process modelling is taken in the EKD approach promoted in Loucopoulos, Kavakli, et al 1997. In this view, EKD is a systematic approach to developing and documenting enterprise knowledge, helping enterprises to consciously develop schemes for implementing changes. EKD advocates a goal oriented view to business process modelling. Instead of imposing a single modelling criterion EKD offers a more general modelling framework that allows several modelling views (or rather modelling components), using the notion of business goals to structure business components in coherent business processes. The above are summarised in Fig. 3 which presents an overview of the EKD modelling concepts. In more detail, a business enterprise in EKD is described as a network of related business processes which collaboratively realise business goals. Business processes are supported by business systems. In the District example the 'customer electrification' process, realises the business goal 'satisfy customer demand for electricity' and is supported by the 'customer information system'.
Business processes are composed of roles that actors (individuals or groups) play in order to meet their responsibilities. An actor is the physical entity (e.g., the 'District technician', or the 'District Technical Section') that plays one or
311
more roles. A role expresses a collection of responsibilities (e.g., 'service providing', 'service administrative handling'~ etc.) and involves a set of activities. For example the 'service providing' role involves activities such as, 'construct meter
customer
installation',
to the e l e c t r i c i t y
'install
metering
device' and 'connect
network').
Fig. 3. Overview of EKD modelling components Activities carried out by different roles deal with business objects; business objects are manipulated by business activities and define the resources and information necessary in order to support the way that enterprise actors fulfil their role. For example the ' i n s t a l l a t i o n ' object is the result of the ' c o n s t r u c t customer i n s t a l l a t i o n ' activity and is described by the following information in the 'customer address
information of
system' : i n s t a l l a t i o n
installation,
town,
town
number, service
code D owner's
name
start and
date,
building
location.
Finally, business processes take place according to a particular logic (or business rules); business rules determine the allowable states of business objects and determine the interactions between different roles. An example of a business rule concerning the installation object is ,wam~ a p p l i c a t i o n form submitted IF contract
4.2
= signed T H E N authorise
construction
of customer
installation'.
Goal-Driven Business Process Modelling
An important aspect of business process modelling in EKD is the representation of business goals. Indeed business processes constitute the means to fulfil strategic business goals. A business process is also seen as a purposeful system in itself. Each role involved in the process intends to achieve one or more defined goals. This does not necessarily mean that every role in a process aims to achieve the same business
312
goal rather that satisfaction of the 'private' goals of individual roles supports the achievement of the business goal that is realised by the business process. Therefore, goals related to a business process present a hierarchical structure whereby individual role goals constitute refinements of higher-level goals that ultimately make up the business goal fulfilled by that business process (see Fig. 4). In this sense business goals not only define but also shape business processes.
Fig. 4. Relation between business goals and business processes
In the example illustrated in see Fig. 4, Rolel :
'service
providing'
role achieves
goal a~,1:'construct new customer installation and connect it to the electricity network'. O n the other hand Role2: 'service administrative handling' role achieves m a n y goals one of which is the goal ~i,2: 'administer servicing of customer' s request for electricity'. Achievement of both goals supports achievement of the overall business goal G0:'satisfy customer demand for electricity' which is realised by the 'customer electrification' process. Thus 'service administrative handling' and 'service providing' roles form part of the ,customer electrification' process.
Business goals do not just shape the current business structure. They also set the vision for business change or business improvement. To this end, business goals establish the context of business change (i.e. the objectives towards which the business change effort is targeted). For example the business goal ' i n c r e a s e District competitiveness' sets the context of business change for the District case. Achieving this goal can be seen as a gradual process which encompasses the causal transformation of the initial goal into one or more subgoals until a plausible business process specification that satisfies the original goal has been defined. In our example the original goal 'increase District competitiveness' Can be refined in the subgoals 'create new markets'~ 'build a commercial profile' and 'improve current functioning'. The latter can be consecutively refined into
313 'improve existing services to current customers' and 'reduce response of any customer request'. T h i s is graphically represented in Fig. 5. Any goal at each refinement level describes WHAT needs be done. At the same time this goal can also be considered as an end (WHY) for another goal, as well as means (HOW) for still another goal at a higher level. time
f,
,e. . . . . . .
I Createn e ~ d a e ~
......
1
w.Y
I ==
......
mereial I
i.z.
Improvecurrent ,mprov. ,
~.................... WHAT i WHY
f tmP'~176176176176 qualit~ J t. . . . . . t eustO . . . .
Operational features
Red. . . . . .
........... f
ponsedmeof any customer request
.ow W~AT'i
)
1
HOW
Fig. 5. Business goals define the context of business change In many cases more than one alternative subgoals can be identified. This will lead to the identification of alternative ways to achieve a business goal and therefore alternative ways of shaping the business. We must note here that goal achievement is not a strict top-down refinement sequence. One could also proceed bottom-up by finding simple goals and then connecting them to higher level ones. Of course, the initial change goals are defined first - otherwise there would be no subject-matter for the whole process.
5 5.1
Applying Goal-Driven Business Process Modelling Relate Business Goal Satisfycing to Process Modelling Strategy
In this section we discuss the empirical results and observations from applying the approach briefly discussed in section 4, to the industrial application (introduced in section 2). Any design task for change normally involves multiple stakeholders and decision makers. One of the aspects of the EKD approach is the support of a reasoning cycle that involves goal setting, deliberation and agreement. Space limitations prevent us from giving a full treatment to this subject but, since it is relevant to the business process modelling activity we briefly describe its use with reference to the industrial application.
314
9
Goal setting consists of establishing the stakeholder goals which designate any objectives to be reached, demand to be satisfied, problem to be resolved, issue to be discussed, etc. in general anything that one would like to achieve in using EKD. Deliberation includes the expression of hypotheses for achieving stakeholder goals (e.g., expressing alternative problem resolutions, making proposals concerning the satisfaction of some demand, etc.) as well as generating arguments for or against such hypotheses. Finally, agreement generates decisions that can alter (produce/modify) the product (the EKD models) while in turn generate new goals to be achieved.
9
9
~-~anAO~,se~s~= to die competition
re-org~t~tl~ requiresa ~ clear view of where the J businesset~ently stends.~
VDEOSON----~ -'Re-focus'businessrolestowards businessprocess= based on the
r-DECISION ~1 obieetivesforehan~e " -- J ---DECISION I . / Relate goals for change to I
q existingbusinessprocesses I DECISION
Use businessgoals for change to identify eritertafor re-designing related
I
business processes
Fig. 6. Reasoning in the District application The benefit from using such an approach is twofold. First, the important components of any deliberation are captured and can be used in tracing the history of the rationale
315
of decisions. Second, it c a n be used as the baseline for evaluating these decisions, relating these to the business goals, business processes and support systems and acting as a framework for quantifying design options. Application of EKD in the District case involved several technical and managerial staff together with EKD experts. An example of the reasoning cycle applied to the way that we approached the business process modelling task is shown in Fig. 6. 5.2
Model District Organisation
Micro-Processes
according
to
Current
Functional
A summary of the business activities performed in each functional section is presented in Fig. 7 which presents a map of District activities as described by the District employees. This map represents a 'vertical' view of the District in which District activities (or rather micro-processes) are organised along the four functional lines introduced in Fig. 2. T e d m k a l Section
C u m i n s " S e r v k ~ Section A I - Elec~:~; SepOy Appllca6~ Fultillment R~-I./V Cmmra:~ A2 - l x l e f ~ 17,esha~ A3 - Meter Discamcc~n A4 - Meter P,c-<~.ec~n A5 - ML'~ O~eck & ~ A6 - tm~lladon Modification A'/- l : a i h ~ ~ A8 - C,m~ Benk Pa)m~t A , ~ o d s a 6 m A g - Revoke B a ~ l ~ m = a At~odsatio. AIO - ~illi~8 Comx~on A l l - Pa~T~r Co~lr162 AI2 - Meter R e a d ~
C u ~ o m ~ E l ~ t r i f ~ . ~ t i ~ Seotio~
C
~
-
H
~
Bl - Hamlia~ O a m ~ c a u ~ o. PPC$ ~ o ~ ~ d~ ~ B2 - Petfocming S~dy of Elocuiclty $ u p ~ . t h w t ~ Mo~fgafion of l . N bkavoA B3 - Perft~ting Study ~ t Ne~. or ExL,t i ~ Mrv"~ B4 - E6 - I~ffom/.g Study of Ekcaldty S , ~ I ~ ~ithout Modificalon of L/V Nea~t~k ~ - P=~i~ming S ~ y on I,~v~ck M ~ i ~ ~ - P-~par of ~ ' s ~o~ ~ LO~ D ~ M ~ or B~c~, out B7 - ~ t ) +Di.~onnoc6on B8 - Ekcaicity gc-comamon [;9 - Network Modi~:atica BI0 - NE'w2t~0 4 KV Sul~tliom ~ BII - New L ~ 20 KV Line (non Affix) ~ B 12 - ~NE-wI]uildi~ ~ fat 20/0 4 K'V ~ B 13 - U/G Cable Re-me,n8 fog 20KV Lines B 14 - Draf6~ ofpmvis~ns ~ a ~ pmgrml m 5 - ~ , c ~ , ~ t ~ fmh~x ,epa~ ~ ~ B 16 - Dtafllag of m a m a , ekoaici~, faiha.~ rcpag ~ k B 17 - Rcpo~ on tat~ts of ~ ~ k sdgduled [$18 - Daily ~ o A ~hedtfle chaffing BI9 - ~ of l o n g - t e n ' a h m ~ wink l ~ m B'20 - P t ~ t r . ~ ~ i n u m ~ wi~ the h e r o d , B21- OSM(FoE M a ; n u ~ e e B22 - ~ n l~emnce ~ - Re.i+ o ~ h a ~ l o u s ~ s B24 - Pcrfcor~g s t u ~ / ~ r ~ k ~
~
I I ~ l l o l m d Sr162 DI - Penomd Trdr~ng D2 - Prevc.6~ o ~ i n d u s ~ a c c i d e ~
B26 - Planning of N c t ~ - k D c ~ o w a = ~ B27 - P e r ~ Study ,~r h ~ open ~ m S - t . ~ a ~ S ~ w u k Ptms B29 - I m p e ~ m of Nod~ of flxs fiG0 - I m p e c a ~ of s/topic insrdla6on B31 - Mmhot pa:,ccssn8 of SAB B32 - ~ r ~ t ~ md Sg~e Pros M ~ t ~ g
B33. E ~
,~&-~omg
~35. w a ~ o , ~ g ~ T ~ B36 - L r Monitofin8 o~ Tram~m~on L/n~ and SOb~fio~ B37 - M o n i ~ I.,'V L m ~ B38 - Menito6ng Mainaman~ of V c h k ~ B39 - l.W, M'V Nev.~o~k Mmitonng
[340- Pzcv~nfv~ ~
of N~.x~k
B42. I-landli~ ~ur r~a~ Dism bu~,~ N c ~ k
Fig. 7. Overview of District micro-processes according to the functional organisation
316
As illustrated in Fig. 7 in the majority of cases District customers contact the company through the Customer Services Section. To fulfil the customer demand there is a need to involve the Technical Section o f the District. The service requested by the customer will be delivered by the Technical Section after authorisation by the Customer Electrification Section. By studying the District micro-processes one can easily conclude that while many activities are performed within different functional divisions they are parts of the same business process. For example micro-process A1:'Electricity Supply Application Fulfilment for the L/V Customers'* and micro-process B3: 'Performing Study of Electricity Supply through modification of the L/V
Network' are parts of a bigger process which deals with the supply o f electricity to District customers. However, this is not obvious in the functional organisation description since there is no description of the interrelationships between different functions. In order to understand the interactions between the functional unit described in Fig. 7 we proceeded to modelling the current District behaviour in terms of actor-role diagrams of District activities. An actor-role diagram presents a high-level view of the association between actors and their different roles. An example of an actor-role diagram for the At: 'Electricity Supply Application Fulfilment for the L/V Customers' is illustratedin Fig. 8.
This diagram describes the actors involved in supplying electricity to a L/V customer. This is a core District activity and is realised through the co-operation of several District actors. This co-operation is modelled in terms of dependencies between roles. There are two parties involved in the dependency: the requester role, i.e. the one that needs something in order to fulfil its responsibilities, and the provider role, i.e. the one that can provide the missing component. This relation can be of various types: (a) authorisation dependency denotes hierarchical dependencies that can exist between roles; the provider role gives authorisation to the requester role, (b) goal dependency reflects the fact that the achievement o f a goal that the role brings about is dependent on the achievement of a goal of another role, (c) co-ordination dependency expresses the need for one role to wait for completion of another role's responsibilities before it can complete its own, and (d) resource dependency illustrates the need for one role to use a resource that can be provided by another role. For example in Fig. 8, the 'service requesting' role depends on the 'service administrative handling' role for the achievement of its goal 'to get connected to the electricity network'. O n the other hand the 'service administrative handling' role depends on the 'service requesting' role for receiving money (which is a resource). Similarly the 'service providing' role depends on the 'service authorising' role for authorising the construction of customer * L/V = Low Voltage
317
installation. Finally, the 'service authorising' role depends on the 'service administrative handling' role for completing the contractual and financialtasks before it can give authorisation (co-ordination).
Customer
f Service %dministrative Handling
Goal:
Goal:
To get connected to the electricity network
Administer servicing of customer's request for electricity
Providing Goal: Investigate all technical parameters
Calculate required materials and Customer Electrification Section
costs
Construct new customer installation and connect it to the network
r
~ Service Authorising
Goal: Authorise service provision to -P the customer
~-
Deal with customer installation logistics
J
Fig. 8. Actor-role diagram concerning the 'Electricity Supply Application Fulfilment for the L/V Customers' The advantage of the actor-role diagram is that it provides a clear view of the interactions across different functional divisions. In this way it becomes apparent that fulfilling a L/V customer application for electricity supply is not solely the responsibility of the Customer Services Section but also depends on the co-operation of the Technical and Customer Electrification Section. Such interactions would appear as inputs/outputs in an activity-oriented view, thus obscuring the fact that 'Electricity Supply Application Fulfilment for the L/V Customers' cannot
be performed independently of other activities performed by other sections. In addition the ability to include the customer role in the actor-role diagram, is a step towards a process-centred view of the organisation in which each process has a customer.
318
Service
Requesting
Service Administrative
Handling
Service Providing
TriggeringEventSubmitapplicationform J. Sendelectrificationserviceorderto TS
InstallMeter L Outcome:Notifycustomer:Applicationfulfille( ConnectMeter Customer Electrification Section I I a. I l /
Customer
CustomerService Section
Technical Section
Fig. 9. Role-activity diagram for the 'Electricity Supply Application Fulfilment for the L/V Customers'
The actor-role diagram gives a f'u'st-cut view of the organisational aspects regarding the responsibilities of individuals or groups in their involvement in the operation of a business process. A more detailed view of these roles was constructed in terms of role-activity diagrams Ould 1995. These diagrams show the set of activities that are generally carried out by an individual or group within the context of their role. An example of a role-activity diagram for the ' e l e c t r i c i t y supply application fulfilraent for L/V customers' is illustrated in Fig. 9. Role-activity modelling encourages the identification of the key operational components which can be measured (activity duration, actor skills, resource costing etc.). In that sense roleactivity models provide the context for performing evaluation of process efficiency.
319
however of"what are the types of processes and how should be logically organised?" still remains unanswered.
I gafisfyeust~
I ~
ooal Zqend
Fig. I0. Partial view of District goals
We address this question by using the business goals that represent the intentions of processes. These goals are presented in the 'body' of each role in the actor role diagram in Fig. 8. Such goals are mainly operational goals, that is they are expressed in terms of business objects and activities that realise them. For example the 'service it
providing'
to the
network'
connect meter
goal to 'construct
refers to the
customer
construct
installation
and
installation, install
connect
meter
and
activities identified in the role-activity diagram in Fig. 9.
As explained in section 4.2 role goals represent low-level goals in a goal hierarchy. Having identified the role goals (i.e. those goals in the goal graph that represent operationalisations of the business objectives) there was a need to establish their causal relationships to higher-level goals. Starting from the goals identified in Fig. 8 we constructed the goal graph presented in Fig. 10.
320 District businessgoals ,~ Satisfycustomerdemandfor electricity Satisfyload ioerea~ ~-4~ Rainforce/extandDistrictnetwork Indemnifyfield ownersfor damagescaused to fiaidsfrom the network ly oastamerwi~ et~a-icity ~
Faeihtataschedulingof rcpalrworks Isolatepart of network for work execonon Scheduleshiftsof failtm~repair taam Execute repairworks 9 Ensuregoodnetworkoperation ~
Preventelecmr
Avoidproblemscaused by contact of network withtree branches Gmnclearpictureof network condition Monitormaterialsand spare partsconve)ancr and performance fadures
~
Ensuregoodoperationof Substations Enstm:goodoperationof network elements Monitorloadand preventoven~harges Improvenetworkquality ~ Modanfiseagriculturalexploitations Planfor network improvements Ensure realisafionof plan targets ~ Resitar net~ork that interfereswith other woP~ in the area
9 Ensurepositivefinancialprofit --~ F ~ p r o ~ t
fromse~vicesprovidadtocastomors Collectdata on customerr co~sla'nption ~ Verifycorrect functioningof metea'ingdevices Facthtata customerpayment ,~ Collectcustomerpayment Stopsuppb of r to non-payingcustomers 9 Resta~ supplyof electrtaityaReroustamer'sdebt Is settled Ensure correotchargingof cus~racl'S I
~ Ensmecorrect calculationof ehichScityconsumpimn
Vorifycon'ctafunctioningof meteriogdr Ices Respondto customercomplaintsconcerningbilhng LI~ Verifyand correct billingreformation Improveexploitationof ocml~myassets t-~ Prolongserwce life of ,,~oodenpoles ~1~ Efficientuse of supportequlpment ~'~ ImprOvenegwOrkexpinltation Improvevehicleexpioitauon Simphfythe process of suparvislagand dis~buting raatarinls ~ - Manageflow ofmatariaisand equipment Train l:hstrictpersonnel Ensure safetyof Dismct Personnel Protaet company's interest ~ - ~ Receiveoom~ls~lon for damages caused tOdistributionnetwork fromthwdp~es ,~ A~oidstealingofelecmcity
Fig. 11. Overview of District business goals
In Fig. 10 it can be observed that goals related to the roles involved in fulfilling the application of a L/V customer for electricity supply, satisfy the higher goal , supply L/V customers with electricity'. This in turn supports the satisfaction of the goal 'supply customers with electricity' which ultimately supports the achievement o f the strategic District goal electricity',
'satisfy
customer
demand
for
321
This process of abstracting from operational goals to higher intentions, naturally involved different stakeholders. The result was a clear, agreed view of the reasons for the business process (the WHY dimension, see also Fig. 5). By repeating this process for all District roles we delivered the following goal hierarchy that explains what the District is currently trying to achieve. A global view of the District goals and the associated activities is presented in Fig. 11. Each leaf goal in this goal tree refers to specific business micro-processes studied in the actorrole/role-activity modelling step. 5.4
Adopt a Process View, Driven by Discovered Goals
Using the goal graph illustrated in Fig. 11 District micro-processes are grouped in five core business processes each aiming to achieve a strategic business goal. These are:
Table I. List of District business processes Business Process
Supply customer with electricity Satisfy load increase Ensure safe and continuous network operation Exploitation and nmintenance of company Improveexploitation of company assets assets Customer Billing Ensure positive profit for services provided to customers
The business processes presented in Table 1 process correspond to the goals highlighted in Fig. 11. Each process is composed by the micro-processes that realise the subgoals of the goal that is realised by the entire process. Thus the map of District micro-processes is re-organised in terms of the goals these activities aim to achieve. The result is illustrated in Fig. 12. In contrast to the 'vertical' map illustrated in Fig. 7, Fig. 12 presents a 'horizontal' view of the District whereby each District process crosses functional boundaries and requires the collaboration of more than one District sections. Indeed it can be seen in Fig. 12 that for each business process there is a horizontal referencing, including micro-processes from two or more District sections (shown as A, B and C and D in Fig. 12). For example the 'customer e l e c t r i f i c a t i o n ' process involves the collaboration of the Customer Services Section, the Technical Section and the Customer Electrification Section.
322
A I - Elec~icity S,u~y A
~
Fulfillment for I~V
m2-pe~aarangSo~yoemm~ys~Cy~
~c,~,
dt;v~ ~ 4 . pn-~nins S ~ y or ~ m m ~ y s ~ y w ~ a ~ Mod~mion o~I./V Net~c~ CI - 13ecnicitySupply A ~ m i o n Ftdfillmen for M/V Omome~
B29 - ~ ofi01ocks ~ n m A,4- ~ R~-oor,n ~ i o n A6 - lm'~laaon Mo~ f ~ , o n A7 - Failure Resxonaico ~ 3 - lvfeter ~
C3 - lmutllation l ~ a m l l m ' ~
Bg- ~ Ivlo~f~tion B5 - l~rfom-a~ Sady on Nn~adCxxfaf~ion BI0 - New 20r 4 KV S d ~ o m ~ BII - New U ~ 20 KV Line (non Attica) C~s~n~tion BI2 - l'4~wI~ldin8 ~ for 20/0 4 KV S'S m 7 - ~nfomm~ s~dy for ~asir~ o~n sa,mtiom B28- Upd~in~ Nero,o@ Rms C2 - I-lm~lir~ fmkl dm-~a~
AI2 - Ma~" R ~ d ~
A5 - M ~ r Cl~r & Ivlninmum~ A8 - Grmt I~1~ Paymm Audmris~on Ag- R_-'~keB ~ k P~mmt A t a h o d ~ A l l - PaVmmtCdlection B7 - ~ t y Disommaion B8 - Eleafiaty l~,~oommk~ AI0 - Billin8 Con'~ion
A7 - Failure Re~or 1~3 - R~o=r @han~-do~ s~maor6 B42 - Hand]ir~ fire n~r ~ ~ B6 - R~p~ of l~n-,a~ to Netv~:~ due to N=und D i s n ~ or Blnck ota BI6- D m ~ of mmthly deemdty failures r e a r v,ork B 14 - D r ~ 5 ~ of ~ w i s i o ~ for a ~ n ' t ~ d pmffam BI 5 - 19~midty failure ml~r by shi~ B20 - Pin.in8 t r ~ that inmfwe with the network B31 - r ~ n i t x ~oce~in8 of SAB B34- t a n e M ~ m i n ~ B32 - lvl~finls md S I ~ l~Is IVlmimring BI 3 - LYGCS~leI~.mtlin8 for 20KV Lines B22 - ~ lVgaimlmame B40 - l%,vo~ve I ~ n t m m ~ or Net~o~ E/onelm B36 - Lind Moni~m~ on T ~ i o n Lin~ md Subm6om BI9 - ~ of ~ n - ~ t t n - n q e n n ~o~k plato BIB - Daily ~ k ~hechle ~ BI7 - Re~xt on t ~ 8 ~ d ' ~ v,ork sc-~duled B2.4- l~donnmg saxly on nm,,~k m~irovm~rm B25 - Oea'orrar~ Sady on a ~ 5 o u l a ~ E l e a r i f ~ t ~ A2 - b,letv,r Resitmon
~ md n~mmee of r B21 - OS/r lvl~nter~-ice m3 - r~pa~t Mmt~i~ m s - w a ~ o u s i ~ and T r m s o ~ , ~ B37- M o o ~ L / V ~ B38 - MonitoringMintmmoe of Vehicles E09 - L'V, M'V lx~v.ork Mmilmin8 Dm- l ~ m d Tr=ni~ 132- l~.v(m6on ~indumial BI - H~xlling D l m l ~ cat~d on !t~C's ~ 1341 - I-~n~lmlgF'lecUidly Stealing
~
by lhird i~'lies
Fig, 12. District process map based on District goals
6
Discussion
In this paper we have presented an approach to business process modelling based on the notion of 'process abstraction through intentional affinities'. The purpose of business process modelling is ultimately the improvement of the enterprise in order to deal with change. In general therefore, we advocate that change management should be seen as the process of identifying business goals and relating business processes to these goals. Returning to the District application, capturing goals for change presented a number of problems stemming primarily from: (a) the uncertainty of the future situation and (b) the different perceptions that different District personnel had on the issues for change. Relating goals to existing processes proved to be the only reliable way to achieve a clear view of the issues for change. The result of this exercise was: (1) to increase the awareness of District personnel about the issues for change; (2) to give the opportunity to strategic and operational
323
District personnel to participate in the creation of the company vision; and (3) to produce a list of strategic goals for change. Relating change goals to existing change processes helps to further refine goals based on the specific business process characteristics. For example 'reduce response time of any customer request' can be refined in terms of measurable goals that refer to specific micro-processes involved in the customer electrification process, e.g., 'reduce p e r i o d from customer a p p l i c a t i o n to D i s t r i c t offer to I0 days',
'reduce
time
necessary
days', and 'reduce p e r i o d
to
construct
from customer
payment
a
new
installation
to m e t e r
connection
to
50
to 40
In addition, the fact that the customer electrification micro-processes have already been modelled in terms of role-activity diagrams, can further assist in evaluating the reasonableness of these proposals and also to reveal the points where process delay occurs and suggesting improvements. days'.
Of course in some cases it is not possible to identify any existing process that can be related to the goal for change. For example 'build a c o m m e r c i a l profile' objective refers to processes that are completely new for a company that did not have to deal with competition up to now. In this case the new process should be defined from scratch again as a causal transformation of business goals (see Fig. 5). The goal of a process organisation is to create a high performance workplace, a high quality work environment noted for excellence in efficiency, effectiveness and customer satisfaction. With a focus on process, it is very common to see process organisations managing interdisciplinary work teams instead of specialised units seen in traditional organisation of enterprises. The approach presented in this paper recognises the need for a co-operative approach to reaching a common view of the products and services delivered by the business processes of an enterprise. In recent years similar approaches have been advocated in the areas of enterprise modelling Jarke, Bubenko, et al 1993;Yu 1994;Kueng and Kawalek 1997 and CSCW Ellis and Wainer 1994. The focus of these approaches is on individual agents and the goals that drive and justify their individual activities as well as their co-operation. Whilst our approach shares this direction, we also advocate a more holistic framework whereby, business goals as well as individual actors' goals are considered in terms of systematically analysing current business processes, the goals for change and the impact of these goals on existing or new business processes. Such a goaldriven, deliberative approach presents, in our opinion, a major step towards meeting this emerging challenge of managing business change.
7
Acknowledgements
The work reported in this paper has been partly supported by the commission of the European Union under the ESPRIT programme. The authors wish to acknowledge the assistance of Mr D. Beis and Mr G. Vgontzas, as well as the participation and
324
collaboration of Professor C. Rolland, Dr S. Nurcan and Dr G. Grosz, in the industrial application described in this paper.
References Alderman, N., Maffin, D., Thwaites, D., Vaughan, A.T,, Braiden, P. and Hills, W. (1997) Understanding customer value: a business process analysis approach, ME-SELA'97, Loughborough, 1997. Bubenko, J. (1994) The F3 Reference Manual,, Deliverable F3, version 0.4, June, 1994. Davenport, T. (1993) The Process Innovation, Harvard University Press, Cambridge, MA, 1993. DeMarco, T. (1978) Structured Analysis and System Specification, Yourdon Inc., New York, 1978. Dobson, J.S., Blyth, A.J.C., Chudge, J. and Strens, R. (1994) The ORDIT Approach to Organisational Requirements, in 'Requirements Engineering: Social and Technical Issues', M. Jirotka and J. A. Goguen (ed.), Academic Press, London, pp. 87-106. Easterbrook, S. and Nuseibeh, B. (1995) Managing Inconsistencies in an Evolving Specification, RE'95, IEEE Computer Society Press, Los Alamitos, California, York, England, 1995, pp. 48-55. Ellis, C.A. and Wainer, J. (1994) Goal-Based Models of Collaboration, Collaborative Computing, Vol. 1, 1994, pp. 61-86. Franckson, M. and Peugeot, C. (1991) Specification of the Object and Process Modelling Language,, ESF Report D122-OPML-I.0, 1991. Hammer, M. and Champy, J. (1993) Reengineering the corporation - a manifesto for business revolution, 1993. IDEF0 (1993) Integration Definition for Function Modeling (IDEFO), Computer Systems Laboratory, National Institute of Standards and Technology, FIPS Pub 183, Dec 21, 1993, 1993. Jarke, M., Bubenko, J., Rolland, C., Sutcliffe, A. and Vassiliou, Y. (1993) Theories Underlying Requirements Engineering: An Overview of NATURE at Genesis, IEEE International Symposium on Requirements Engineering, IEEE Computer Society Press, San Diego, California, 1993, pp. 19-31. Kueng, P. and Kawalek, P. (1997) Goal-Based Business Process Models: Creation and Evaluation, Business Process Management Journal, Vol. 3, No. 1, 1997. Loucopoulos, P., Kavakli, V., Prekas, N., Rolland, C., Grosz, G. and Nurcan, S. (1997) Using the EKD Approach - The Modelling Component, UMIST, WP2/T2.1/UMIST/1, April 1997, 1997. Ould. M. (1995) Business Processes: Modelling and Analysis for Re-engineering and Improvement., John Wiley & Sons, Chichester, 1995. Ross, D.T. and Schoman, K.E. (1977)Structured Analysis for Requirement Definition, IEEE Transactions on Software Engineering, Vol. SE-3, No. 1, 1977, pp. 1-65. Singh, B. and Rein, G.L. (1992) Role Interaction Nets (RINs): A Process Definition Formalism, MCC, Technical Report, #CT-083-92, July 1992, 1992. Swenson, K.D. and Irwin, K. (1995) Workflow Technology : tradeoffs for Business Processes Re-engineering, Conference on Organisational Computing Systems COOCS 95, CA, 1995. Yu, E. (1994) Modelling Strategic Relationships for Process Reengineering, Ph.D., Universityof Toronto, 1994.
Real-Time Information System for Risk Management on Motorways Tullio Tanzi 0'2), Sylvie Servigne(1), R6gis Guiol ~3) tl~Laboratoire d'Ing6nierie des Syst~mes d'Information, INSA de Lyon, 20 Avenue Einstein, Villeurbanne Cedex F-69621 servigne @if.insa-lyon.fr <2)Etablissements Jean GRANIOU, ZI des Trois Moulins, BP 717 Antibes Cedex 1 F-06633 <3)Direction Recherche & D6veloppement ESCOTA, BP41, Mandelieu Cedex F-06211
Abstract. Every day more and more people and goods have to be transported
rapidly, safely and at a lower cost. Continuous economic growth brings a traffic volume increase and implies a gradual saturation of road and motorway networks. Managing motorways is today a complex task, especially during a time of crisis (traffic jam, accident). Our aim is to design a system for risk management on motorways. Risk management is a very important endeavour today and many people work on this subject Adams 95, Beck 92. In our system, risk management provides tools to help managers to anticipate potential accidents so as to avoid them or, otherwise to reduce accident consequences. In addition, the system must be able to give fit information to initiate preventive actions and then to follow them in real-time. Such a system requires much information in real-time. Information must be acquired and then sent to the various motorway departments to be processed. It is necessary to have a complex real-time architecture based on sensors and communication technology to link motorways with the management stations. The proposed global system for risk management on motorways is presented in this paper. Real-time information composition and use particulars are presented. Details on processing can be found in Tanzi 97b. An industrial prototype has been realised and for the motorway network of the south of France.
1
Introduction
In managing toll motorways a complex chain of decisions has to be taken by operators when an accident occurs in order to rapidly restore to a normal traffic. Indeed, even a minor incident (the fall of luggage from a car, a small collision) can have consequences that hold up the traffic during several hours after the end of the incident Boulmakoul 93. The number of cars caught in the resulting traffic jam depends on the time needed to repair the carriageway. The delay between the incident and the arrival of intervention vehicles implies increasing difficulties in accessing to the incident location because of the traffic jam Tanzi 97a. As lanes are one-way lanes, it is very difficult to use lanes in the opposite direction, even for vehicles authorised to do so. To come back to a normal situation, it is then necessary to evacuate the cars caught in the traffic jam, which may last a long time. Various studies on traffic control show that
326
a reduction of 30% of the total intervention duration implies a reduction of 55% of the lost time of the drivers HCM 95.
To obtain a realistic overview of the situation on the motorway, motorway managers receive traffic data from measuring sites at their disposal. Measuring equipment are numerous and set along the motorway at frequent intervals. To make an optimal use of traffic data, it is necessary to send them rapidly from an acquisition station to the processing station and to assemble these data from several acquisition stations to produce a global overview of motorway areas. Improving the speed of data transmission allows operators to react rapidly to incidents. It is so vital to have a realtime system. Depending on the type of incident and existing information concerning it, it could be possible to anticipate the potential accident so as to avoid it by initiating preventive measures. Nevertheless, if it is not possible to avoid accident, preventive actions could still reduce or ameliorate consequences.
Our aim is to build a system for risk management on motorways. A sine qua non condition for a system able to be successful in preventing motorway risk is to obtain immediately information which characterise traffic and environmental conditions. The aim of the paper is to present our proposed architecture for a real-time system. First, information characterising traffic will be described. Then the real-time system will be presented. After a general presentation of the system context, information flow and processing will be described, going from the acquisition process to their use in the decision phase. The prototype has been realised on the Escota's motorway network. The Escota company is in charge of a part of the motorways of the south-east of France.
2
Traffic Data
Traffic is characterised by a flow (distribution of vehicles in time), an business index of a given road segment (concentration) and the vehicle speed. Different sensors are used by traffic managers for traffic data acquisition. The most common procedure for traffic data supervision consists in using magnetic loops buried in the road surface, which detect the passage of metallic masses (vehicles). Two loops situated successively in the same lane (Figure 1) allow acquisition of vehicle speed.
327
Fig. 1. Calculation station
The processing of the data produced by various pieces of equipment that already exist on the motorway network gives an image of the real situation. The Figure 2 shows an example of traffic properties of a given motorway lane.
Station 23 23 23 23 23 23 23 23
1 I 1 l I 1 I I
Weight 0 0 0 0 0 0 0 0
Time Dam 20/04/97 00:30:00 20/04/97 00:36:00 20/04/97 00:42:00 20/04/97 00:48:00 20/04/97 00:54:00 20/04/97 01:00:00 20/04/9701:06:00 20/04/97 01:12:00
Flow 800 800 720 740 700 590 620 630
1 1 I l 1 1 I 1
Speed I 15 112 I 13 112 108 114 III 108
I I I I I 1 I 1
Oe ram ! 1 I 2 I I I l
1 I 1 I I 1 I l
Truck Flow 0 5 3 5 0 I 0 13
I I I I 1 I I 1
District 11 I1 lI I1 II 11 I1 II
Fig. 2. Example of real traffic data of a lane
The processing of such data into a display form allows the preparation of three curves (speed, flow, and concentration) as seen in Figure 3. In this example, a first accident occurred at 10h40 am, and a second one occurred one hour after. Nevertheless, an inspection of these curves suggests that the risk of the first accident can be detected as early as 10h00 a.m.
328
iU. ---.. i. . . .:.----.l. . ................ .t. ---.. l. . . . i.
Fig. 3. Display of traffic data Our aim is to use the delay between the risk detection and the potential accident to avoid this accident by implementation of a preventive action. For example, the vehicle speed may be reduced by displaying messages on dynamic roadsigns along the motorway. The most important action is to alert the motorway so that they can anticipate the potential crisis situation. More details concerning risk rate estimation may be found in Tanzi 97b and Oppe 90.
3
Real-time system for traffic control
More and more sophisticated equipment constituting a global information acquisition system are available on motorways. It is necessary to adapt data exploitation tools to this context. This can be done taking real-time data acquired by sensors of the traffic management system into account. Then data are used for various purposes at various levels of the hierarchical motorway organisation (Technical Office, District, Control Station) in order to produce a synthesis to help managers to make a decision. Between the acquisition of data and its use for a decision, it is necessary to implement a data processing sequence. 3.1 I n f o r m a t i o n use
The various informations acquired along the motorway may be used for different purposes and by different departments (Figure 4). First, information is used by departments in charge of the viability of the network in order to maintain the quality of level of services. Information is also used during a crisis situation when an incident occurs, to verify if the proposed solution is appropriate to the incident. This is possible using people knowledge about conditions about the accident site. Strategy definition for the management and intervention by operators is easier using these
329
information. After the crisis, the chronological account of computing events is used for training, enriching the management organisation experience using the details of the stored information. In future, these data could be analysed and taken into account into the development of long-range action plans so as to avoid accidents and to manage accident situation better. Resources Matching
> Str.eo Service
Definition
G
Training
Fig. 4. Use of acquired information To understand and simulate traffic variations, numerous parameters must be taken into account at the same time because of data and context variety, but also due to mechanism complexity. It was necessary to design a hierarchical method so as to enhance the impact of parameters on the global process. The resulting method allows information filtering. All non significant-information (low impact on the phenomenon) is rejected. The information kept depends on a threshold value which has to be determined according to a desired precision.
3.2 Risk management At the beginning, an accident is often slight and controllable by the appropriate person, at the right time and at the right place. Concerning a motorway incident, the fit persons are the staff in charge of motorway. Telecommunications and computers allow decisions at a distance by means of a virtual visit to the right place. But what about the right time ? In the risk domain, a little problem may generate a lot of trouble beating in mind that time operates against humans. We define the right time as the closest time to the event occurence, and, if possible, the best fit time would be the time just before the event. When an accident analysis is done a posteriori, we can realise how important event anticipation is, that is to say, to be at the right place, at the right time with the appropriate intervention measures. We chose to work on event prevention. We define prevention as the technical measures already located on the site when the incident occurs. The aim of prevention is to prevent from common risk and to limit the accident and its consequences.
330
3.3 Real-time decision process
When an incident occurs, real-time information allows verification of how appropriate resources are for the problem. Real-time data simulation tools gives a projection of the incident evolution and so allows identification the gravity of the incident. Action plan implementation and strategy definition are also based on the real-time data. Figure 5 shows the event management in real-time. Recording information also facilitates building the incident memory and enriching the organisation experience. In the future, modifying generic intervention plans by this experimental data will improve an operators ability to confront an accident.
data acquisition
I
0 D Actions
momo
Fig. 5. Event management in real-time
The decision process is composed of steps (Figure 6) which are not always sequential. Sometimes, backtracking is necessary. The first step is the relevant data acquisition. The aim is to collect as much information as possible concerning the problematical event that occurred. Next, the conception phase consists in developing one or more a priori possible solutions and scenarios. At this step, additional information can be required which implies a second acquisition process (backtracking). The third step consists in choosing the best solution among the list of alternatives previously built. The final step, the evaluation process, consists in evaluating the previous choices and decisions. The aim of the final step is to correct if necessary the entire reasoning.
331
]Acquisitionl~ [ Design I ~ ]
Choice [ ~ l
Evaluation[
Fig. 6. Decision process To validate the proposed architecture, a prototype has been made on Escota's motorway. The prototype is devoted to help the operator follow the evolution of conditions which can imply an accident occurrence and so to detect automatically geographic areas where the risk of accident is high. All information and calculated data are displayed in real-time and continuously. Data are issued from acquisition stations (Figure 7) and meteorological stations. Other information, for example concerning road works maintenance, is acquired through communicating with an existing information system.
Fig. 7. Data acquisition stations
4 Architectureof the real-timesystem In analysing traffic data, the automated system may detect conditions which are going to favour the increase of the risk level of every motorway segment. When a threshold of the risk is passed, the system is setting into a state of pre-alert. Information (motorway synopsis, flow, vehicle speed, concentration (Figure 14)) are displayed automatically and continuously on a specific screen. The risk index is computed and displayed in real-time. The operator is able to follow in real-time the evolution of the conditions and so to decide what to do to prevent or reduce consequences of a potential incident. To realise these operations, a specific architecture is needed.
332
Fig. 8. System architecture along the motorway Figure 8 gives an example of a potential global system architecture for a motorway. This architecture requires use of information issued from traffic data instruments at various levels of the hierarchical motorway organisation (Technical Office, District, Control Station). Information flow and according processing offices are graphed in the Figure 9. The foundation consists of Acquisition Stations in charge of data acquisition. Then, Technical Offices have to concentrate these data. A local analysis is undertaken at the district level. Finally, data consolidation is made by the Control Station.
Conso~ Local A n a l y s i s ~ ,/~nntrnl~ Concentratio~ / A N / ~'~'~;'~"
Acquisi '
U
o
~
Fig. 9. Information flow of motorway organisation
333
4.1 Data acquisition Data acquisition is realised in real-time by sensors (as previously described in the first part) and characterises each passing vehicle. Acquisition Stations are in charge of realtime acquisition and then data transmission to the Technical Offices upon which they depend.
4.2 Data concentration Concentration of acquired data is realised by various Technical Offices. Each station corresponds to a specific geographical area of the motorway. First, data are stored locally and sent every night to a the control station by batch processing. The chaining processing is described in Figure 10. Upon arrived at the Technical Office, data are analysed to produce a synthesis of every Acquisition Station situation. Meteorological data are integrated here into the analysis process. As previously said, processing depends on climatic conditions, for example, a safe distance between vehicles depends on road-grip characteristics. Meteorological data may be furnished by a local station or acquired from a distant meteorological data server.
TrafficSensors ~ ,
Meteorological_
Agregation " ' - - - . ~ ~
Cr~atC~si /
Fig. 10. Data acquisition : processing sequence Depending on the type of acquired data, a pre-alert (low-level) or alert (high level) situation may be generated for every acquisition station federated by a Technical Office. When a pre-alert or alert situation is detected, the data transmission process is realised. It consists in transferring data issued from every Acquisition Station to the corresponding district. Transferred data are raw data (acquired data) or aggregated data (for example, for a period of one or six minutes). The choice of the most adequate data is done according to the alert level of every Technical Office.
334
Generally, data are transferred every six minutes during a pre-alert phase and every minute during an alert phase. It can be changed by intervention by a district operator.
4.3 Data local analysis The local analysis is made by Districts. A District is an administrative entity representing about 40 kilometres of motorway. Every District receives a data synthesis from the Technical Offices. A cyclical observation of the traffic is made for every station. The frequency of data scanning depends on the alert level of every station. Figure 11 presents the processing sequences for the traffic observation.
z73
Basis I ..J Calculati~
bservationI V rramc E~S~ }
,YES
O
,F I Pre-a
7oUL s
I TI ~-~1Y~s NO.or
I
I Oporato,
I Actions I
Fig. 11. Traffic observation processes The traffic observations allows in real-time verification of the evolution of the traffic conditions. The display given as figure 12 shows an interface of our prototype during supervision time.
Fig. 12. Supervision Time interface
335
When index values reach a defined threshold, the system moves into a pre-alert phase and automatically switches the information onto a specific screen. This operation is devoted to attract attention of employees. When a second threshold value (more important than the first one) is reached, an alert phase is initiated. Data acquisition and calculation (risk index) are real-time and continuous processes so information are always displayed. The value of the alert threshold has been defined by various surveys of motorway companies, and estimated at 90% of the maximal capacity of the traffic. To keep enough time for the pre-alert phase, it is important for the first threshold value to be lower than the alert threshold (usually between 10% to 15% less). Figure 13 shows our prototype interface for an alert phase. In addition to the various indexes already represented in figure 12, numerous windows that allow to following the data evolution in detail are displayed. Every window corresponds to one data acquisition station. When a incident occurs, managerial operators of the incident can see on the screen the impact and evolution of the event on the traffic.
Fig. 13. Alert phase interface 4.4 Data consolidation
Data consolidation is done by the Control Station of the motorway. Information concerning road works on the motorway are taken into account by the Control Station. To do this, the system uses data issued from various external databases belonging to different departments of the motorway, like the road works planning department.
336
Taking into account road-works information, the Control Station is able to balance synthesis elaborated by districts. The Control Station has a unique and global vision of the entire network of the motorway at it's disposal. Nevertheless, the Control Station can have a localised display of one or more districts.
Fig. 14. Architecture of Exploitation Support System for motorways Figure 14 represents a classical architecture of an Exploitation Support System for motorways. A "risk server" is added to complete the existing system In this case, system interoperability was realised based on classical database management systems and XWindows capacities for interface conception.
4.5 Preventive actions The risk rate estimation of infrastructure allows the creation of a strategy in order to reduce incident consequences or, if possible, to prevent the event. It is important to detect specific positions where the traffic is quite saturated with high speed vehicles and too short distance between vehicles to avoid a "pile-up". When such a position is detected, the target of a preventive action is to make drivers reduce their vehicle speed. This can be done using electronic boards displaying dynamic messages along the motorway. It is also possible to prevent traffic congestion by lane stop using an automatic folding arm (BRA [Guiol 94] : "Biseau de Rabattement Automatique") as already exists on the Escota's motorways.
5
Conclusion
As more and more people utilise motorways, the risk of traffic jams and accidents increases. When an accident occurs, it is very difficult to obtain information and to
337
intervene rapidly. The aim of our system is to help motorway managers in real-time to prevent accidents. If it is too late for prevention, then it is valuable to help them to build their intervention strategy and to follow their implementation. Today more and more electronic equipment along motorways and the substantial data produced bring out communication difficulties. It is important to organise processing as parallel processing in order to have a real-time working situation. The aim of the architecture of the real-time information system presented was to optimise the communication network capacity. Only useful data (concerning raw data or data synthesis) are sent on the network. Only appropriate data are given to operators if a pre-alert or alert situation is detected. The data transfer depends on the motorway situation (normal : no transmission, pre-alert and alert : transmission) and the system moves automatically from one state to another according to received and calculated data. Data are permanently displayed and, for efficiency, are graphical. The real-time aspects of the system allow people to react rapidly and sometimes to anticipate a potential crisis. A risk index has been elaborated to detect traffic conditions (normal, pre-alert, alert), it was not presented in this paper. More details can be found in Tanzi 97b. The experimental prototype checked the feasibility and coherence of the proposed system (including risk index). To do this, we achieved by setting up simulations using fictitious data selected with the aid of the operations staff of the Escota Company. Then we set up a control set consisting of simulated actual situations. We were thus able to estimate the accuracy of the system's reaction. Now we are working on an integration of our system into an Exploitation Support System as it is used in the motorway domain.
References Adams 95 Adams J., Risk, University College London Press, London 1995. Beek 92 Beek U., Risk Society, Ed. Sage, London 1992. Boulmakoul 93 Boulmakoul A., Selam S. Intelligent Intersection : Artificial Intelligence and computer vision techniques for automatic incident detection. Artificial Intelligence in Traffic Engineering, 1993, VSP International Sciences Publisher, Zeist, the Nerdherlands. Cohen 90 Cohen S. Ingtnierie du trafic routier, Presse de I'ENPC, 1990. Guiol 94 Guiol R., Neutralisation automatique de voies rapides et autoroutes. Revue des Associations des Ingtnieurs et Anciens El~ves de l'Ecole Nationale des Ponts et ChaussEes, PCM LE PONT n~ Dtcembre 1994, p. 25-27 Oppe 90 Oppe S., Koostra M. J., A mathematical theory for related long term developments of road trafic and safety. Proceedings of the 11th International Symposium on Transportation and Traffic Theory. New-York, 1990. p.89-99 Tanzi 97a Tanzi T., Servigne S., A Real-Time GIS for Accident Prevention on Toll Motorways. Proceedings of JEC'97, Joint European Conference an Exhibition on Geographical Information, Vienna, Austria, April 16-18, 1997. p. 42-50
338
Tanzi 97b Tanzi T., Guiol R., Laurini R., Servigne S., Risk Rate Estimation for Motorway Management. Proceedings of TIEMEC'97, The International Emergency Management and Engineering Society, Copenhagen, Denmark, June 10-13, 1997. p. 125-134 HCM 95 Transportation Reseach Board. Highway Capacit Manual, Special Report 209, US Transportation Research Board, Washington D.C.. 1995
Describing Business Processes with a Guided Use Case Approach Selmin Nurcan, Georges Grosz, Carine Souveyet CRI, Universit6 de Paris I Panth6on-Sorbonne, 90 rue de Tolbiac, 75013 Paris, France email : {nurcan, grosz, souveyet}@univ-parisl.fr Phone : +33 (0) 1 40 77 46 34, Fax : +33 (0) 1 40 77 19 54 Abstract : Business Process (BP) improvement and alike require accurate descriptions of the
BPs. We suggest to describe BPs as use case specifications. A use case specification comprises a description of the context of the BP, the interactions between the agents involved in the BP, the interactions of these agents with an automated system supporting the BP and attached system internal requirements. Constructing such specifications remains a difficult task. Our proposal is to use textual scenarios as inputs, describing fragments of the BP, and to guide, using a set of rules, their incremental production and integration in a use case specification also presented in a textual form. The paper presents the structure of a use ease, the linguistic approach adopted for textual scenarios analysis and the guided process for constructing use case specifications from scenarios along with the guidelines and support rules grounding the process. The process is illustrated with a real case study borrowed to an Electricity Company. Keywords : Business Process Description, Use Case Specification, Textual Scenario Analysis.
1 Introduction A Business Process (BP) is defined by Hammer and Champy in 8 as a set of activities which produces (from one or several inputs) an output valuable for the customer. For the sake of improving or re-engineering or simply understanding BPs, Hammer and Champy consider essential to start to describe them as accurately as possible. A BP can be described at different levels, each level corresponding to different types of BP requirements. First, a BP can be described with a set of interactions between agents involved in the BP, we call these interactions <>. In such a description, the IS is considered as an agent. The BP description can be further refined and completed by adding the requirements of the ~ system internal >>.These levels of description are summarised in figure 1. Similarly to Jacobson 9, we believe essential that "A tight, seamless relationship is required between the process that develops the business model and the process that develops the information system". Therefore, we consider that the development of the
three levels of description must be considered seamlessly. Finally, the modelling technique to be used for describing BPs shall be forceful in that it should be possible
The work presented in this paper is partly supported by the European Community within the framework of the ESPRIT LTR project CREWS (Co-operative Requirements Engineering With Scenarios) n~
340
both to show easy-to-understand surveys of the BP and to describe parts of the BP in details [ 10], this complies with the different levels of description we propose.
Fig. 1. Three levels for describing business processes On the one hand, we propose to describe BPs as use cases. As mentioned by Jacobson [10], use cases are a simple, natural way to identify and describe BP. Indeed, use case models are interaction oriented models that focus on the communications between the agents of art organisation. They are therefore very well adapted to the two first levels of description presented in figure 1. Complementary, use case driven approaches [2, 3, 9, 10, 19] have proved useful for requirements elicitation and validation. A use case is a description of one or more end to end transactions involving the required system and its environment [17]. The basic idea is to specify use cases that cover all possible pathways through the system functions [3, 19]. On the other hand, it is not sensible to imagine that a description of a BP can be obtained in one shot because BP are complex and involve many agents. Therefore, our proposal advocates for an incremental process for the description of BP. The incremental process guides, using a set of rules, the description of fragments of the BP and their integration into a single use case specification that includes all levels presented in figure 1. Therefore, a use case specification describes the context of the BP, the structured description of all interactions between agents involved in the BP (including the IS supporting the BP) and requirements about the IS. Finally, we propose to use scenarios as a means for describing the fragments of BPs. In both the Software Engineering (SE) and the Human Computer Interaction (HCI) communities scenarios are widely used as 'engine of design' [1, 15]. In the HCI community, they are used to elaborate on usability issues [7, 16, 26] and to help the expression and understanding of the goals of the design [12, 14, 22]. In the SE community, scenarios serve mainly as a front end to object oriented design [3, 20, 23, 24, 27]. Scenarios can be expressed in various ways : text, animation, prototypes, etc. but textual ones are recommended by several authors [5, 10, 13, 16, 18]. Natural language provides a simple means for describing a problem [12]. However, the lack of guidelines to support the process of constructing use cases is certainly one of the major drawbacks of use case driven approaches for requirements engineering [25]. In this paper, we propose an approach to guide textual use case authoring for the description of business processes. Our approach is an extension of the work described in [21] where the use case specification process was limited to the study of system
341
interactions. We enlarge the scope of a use case for the description of the interactions between the agents of an organisation participating to a BP and therefore to describe what we call a "rich" use case. The input of the guided process is a set of textual scenarios describing the BP. The output is a use case specification of the BP, including organisational interactions, system interactions and system internal requirements expressed in a structured and non-ambiguous natural language text. Indeed, part of our approach relies on natural language analysis. Between the two extreme of using too constraining clauses templates (e.g. 2) and completely free mode of expression (that increases the risks of ambiguity, inconsistency and incompleteness and makes automatic interpretation difficult), we chose a middle one. We propose to combine the use of narrative prose to express scenarios with structured natural language for the construction of "rich" use case specifications. The remaining of the paper is organised as follows. The use case model is briefly presented in section 2 along with its semantics, its associated linguistic patterns structures and an overview of the guided process. In section 3, we illustrate the process and the use of the rules with a real case study of a business process called "Electricity Application Fulfilment" (EAF) borrowed to an Electricity Company (EC). Finally we draw some conclusions and identify future work in section 4. 2 Overview of the Approach Central to our approach are the structure of a use case, the linguistic approach and the rules which ground the process of use case authoring. These three elements are described in turn. More details and examples about this approach can be found in 21. 2.1 The Structure of a Use Case Because of the complexity of the use case structure, our approach proposes to construct use case specification incrementally taking into account partial stories called scenarios. These scenarios are the inputs provided by the use case author according to the structure depicted in figure 2. Context
9 Name 9 Goal 9 Initiating agent 9 Initial state Initial scenario
i +o +1 Final state
Normal scenarios
Exceptional scenarios I " Occurrence conditions
iI I'ooo+onoe++ons I I
9 Path of actions ~
I+a--
LA i I.oco+ .oe conditionsI I 9 Path oractions
+a'+
.l
"
Fig. 2. The structure of the textual inputs of the guided process In this description, there is first a contextual information which role is to situate the business process in its organisational setting (what Jacobson calls <>in 10. The context comprises the name of the BP, the description of the initiating agent (of the BP) who has a goal in mind. For instance, in the case of
342
the EAF process, the initiating agent is the customer of the EC whose goal is to be connected to the EC network. There are also some conditions required for the process to begin. For instance, the interactions of the EAF process does not start if the following condition does not hold : "the customer is in the EC office with a completed application form". Such conditions describe the initial state of the use case and are expressed in terms of agent or resource states. A resource can be physical (e.g. an identification paper) or abstract (e.g. a customer record). After having provided the contextual information, the use case author provides all possible scenarios. We distinguish two types of scenarios: normal and exceptional. A normal scenario describes one (possibly conditional) course (path) of actions reaching a final state that fulfil the goal of the initiating agent. An exceptional scenario is also described as a course of actions. However, the final state of an exceptional scenario does not fulfil the goal of the initiating agent. Actions can be either ~ system interactions )) if one of the involved agent is the system supporting the BP or ~ organisational interactions )) if the system is not involved. An action may be an atomic action or a flow of actions. An atomic action materialises an interaction from an agent to another agent and requires some resources. The sentence "the customer requests the commercial employee for a connection to the network" is an example of atomic action from the customer agent to the commercial employee agent. The parameter of this action is a resource ("a connection to the network"). We consider two types of atomic actions: communication actions between two different agents and internal actions involving a single agent. The previous example of action is an illustration of a communication action whereas the sentence "a technical employee performs the connection to the network" is an example of internal action. We distinguish four types of communication actions: service request, service provision, information request and information provision. In a service request action from A to B, an agent A asks a service to an agent B. Complementary, in a service provision action from A to B, an agent A provides a service to an agent B. "The customer requests the commercial employee for a connection to the network" is an example of service request action which is satisfied by the service provision action "the customer is informed that the connection to the network is done". An information request is a communication action where an agent is asking for some information. "The commercial employee asks the customer to sign the contract" is an example of information request which expects the performance of the information provision action "the customer gives to the commercial employee the signed contract". All scenarios are incrementally integrated in a use case specification. The purpose of the use case is to describe how the initiating agent can interact with other agents to achieve his/her goal. The internal representation of a use case specification is a set of episodes (see figure 3). An episode comprises aflow of actions and the corresponding final states. There are several possible final states for an episode and therefore, the flow of actions can include several paths to reach each of the possible final states. A flow of actions is a complex action composed of other actions, it is similar to the flow of events as defined by Jacobson 10. The composition is based on the sequence, concurrency, iteration and alternative constructors. "The customer requests for a
connection to the network, then the commercial employee asks his identification
343
papers and the location of the house to be connected" is an example of a flow of actions comprising a sequence of two atomic actions (request and ask). "If the meter exists, a technical employee performs the connection to the network" is an illustration of an alternative flow o f actions. Flow conditions are necessary to integrate several courses of actions in one complex flow of actions. In the last example, the flow of actions integrates the description of what happens when the condition "if the meter exists" is ttue. We distinguish two types of episode : normal and exceptional. An episode is said normal when each of its possible final states ensures the fulfilment of the user's goal else it is said exceptional. An exceptional episode describes a <<non normal >>course of actions reaching a final state which does not fulfil the goal of the initiating agent. In the EAF example, the normal episode corresponds to the flow of actions ending to the connection of the customer to the network. "If customer record is written-off' starts the description of a non normal course of actions described in an exceptional episode. An exceptional episode has an occurrence condition and includes a reference to an action of the normal episode in which the occurrence condition can become true. A use case specification comprises a single normal episode which is the integration of all normal scenarios, and a set of exceptional episodes that are associated to exceptional scenarios. A use case specification may be connected to system internal requirements as described in the third level of figure 1. System requirements other than the ones which are formalised in a use case specification itself may emerge in the course of the specification process. "EC must have the land register information" or "EC must maintain the list of written-off customers" are examples of such system intemal requirements. As mentioned earlier, all scenarios are provided in a textual form. Furthermore, a use case specification is also expressed in textual form. Consequently, there is a relationship between the text structure and the use case structure. The text structure can be decomposed into more elementary structures which are either clause structures or sentence structures. For example, the text "the customer requests for a connection to the network, then the commercial employee asks his identification papers and the location of the house to be connected" is a sentence structure decomposed into two elementary clause structures corresponding to the clauses: "the customer requests for a connection to the network", and "the commercial employee asks his identification papers and the location of the house to be connected". Sentence and clause structures correspond to the surface structures of the textual specification. They have a meaning which is respectively provided by sentence and clause patterns. The deep structure of the use case specification is provided by the sentence and clause patterns. Sentence patterns provide the semantics of sentences expressing sequence, conditional flows, etc. Clause patterns give the semantics of actions such as service provision actions, information request actions, etc. Sentence and clause patterns which are case patterns in a case grammar are presented briefly in the following section.
344
2.2 The Linguistic Approach The approach of use case specification presented in 21 is based on Natural Language (NL) text. There is thus, a necessity for catching the semantics of text. In order to fill the gap between the informal representation and the formal model of use cases, a Case Grammar 6 is used. It focuses on the notion of action which is central in the use case model and permits to catch both the semantics of NL and the semantics of the use case model. Following Fillmore's approach 6, the case grammar introduces a set of semantic cases such as agent, object, destination, etc. Semantic cases define the semantic roles that the different elements of an action clause play with respect to the main verb. Semantic patterns are defined to associate a semantic case to the different elements of clauses and sentences. The purpose of the semantic patterns is to define the semantics of the clauses and of the sentences which are respectively expressed in the atomic actions and the flows of actions of use case specifications. At the level of clauses, case patterns are clause semantic patterns associated to verbs. Action clause semantic patterns provide the semantics of the atomic actions of the use case model by associating a semantic role to the related agents and parameter objects. State clause semantic patterns provide the semantics of object states on which rely the initial and final states and the flow conditions of flow of actions. Clause semantic pattems are presented in the form N (V) C, where N is the name of the pattern qualifying the intrinsic semantics of the verb V, and C is the list of cases to be associated to the elements of the analysed clause. To represent the semantics of a clause in a use case specification consists in instantiating a clause semantic pattern. Identifying the concepts of the use case model from natural language is a two stage process which requires first the semantic analysis of the text and second the mapping of the resulting semantic patterns onto the concepts of the use case model. These two stages are illustrated in section 5 which describes more extensively the process of constructing the use case specification of the EAF example. Details and examples about the approach we use for natural language analysis can be found in 21. 2.3 Guiding the Use Case Specification of Business Processes The process of use case specification of business processes is a stepwise process which guides the progressive transformation of input prose texts (starting with the initial scenario) into refined and structured texts and their integration in the use case specification. It comprises four main steps to : 1. define the context of the use case, 2. describe and complete the initial scenario, 3. integrate the scenario into the use case specification, and 4. prompt the need for new scenarios and guide their description and integration. During step 1, the use case is situated in its context by defming its name (the business process it describes), the initiating agent, the goal of the initiating agent and the initial state. Step 2 is an iterative one. It starts with the capture of the initial scenario. It is a text describing a course of actions that can be incomplete and ambiguous. It proceeds with the check and possible interactive completion of the initial scenario. The result of step
345
2 is a complete description of a pathway in an episode expressed unambiguously according to our semantic patterns. It corresponds to one path in the graph of episodes of the use case. During step 3, the completed scenario is integrated into the episode structure of the use case. Positioning the scenario in the use case specification is inferred from the study of the flow conditions. Performing step 2 and 3 may prompt the need for new scenarios (step 4) leading to iterate sequences of steps 2, 3 and 4. At this stage, a shift in the levels presented in section 1 (see figure 1) can be performed. For instance, as it will be shown in the next section while presenting the example, it is possible to shift from the description of "organisational interactions" to the description of "system interactions". The same applies to "system internal" requirements. The shift could also be the other way around (i.e. from a "system interactions" to "organisational interactions"). Guidelines and rules are deemed to support the performance of each step. Guidelines help the author to perform the requested action. Rules define how to map textual scenarios onto the internal use case structure and to reason about it. The overall architecture of the process is sketched in figure 3. The front end is a guided communication with the user. The back end is based on rules working on the use case structure. i
Inlcrnal Representallon o f the Case Specification i. . . . . . . . . . . . . . . . . . . . .Use .......
9
Fig. 3. Overall architecture of the process.
Guidelines have the form of plain text which can be prompted to the use case author on demand while writing down a scenario. They provide recommendation on the style of writing narrative prose. They suggest, for example, to avoid the use of anaphoric references, of synonyms and homonyms, ways for expressing an action, a conditional action, etc. They also advise the author on the expected contents of his/her prose. For instance, a guideline states that "The expected scenario prose is a description of a single course of actions. Alternative scenarios or exceptional treatments are described separately", etc.. There exists guidelines specific to levels 1 and 2 mentioned in section 1 (see figurel). For example, all interactions described at level 2 involve the computerised system supporting the business process as one of the agents. Rules are of five types : (1) analysis rules analyse the semantic contents of each sentence against the linguistic patterns; (2) mapping rules map the elements of instantiated patterns into elements of the use case model, (3) refinement rules include the clarification and completion (3) integration rules help in positioning the scenario
346
in the use case specification and (4) e m e r g e n c e rules prompt the need for other scenarios. Below, we exemplify the different types of r u l e s mentioned above. More will be introduced in the next section with a walk through the EAF example guided process. All rules have a premise and a body separated by a <~---~>>. They are described by a first order logical formula. The premise defines the precondition for executing the body. This precondition is expressed on elements of the use case specification and defines when the rule is applicable. The body is either an action required from the author (using the ASK predicate) or the generation of new elements of the use case specification (using the GENERATE predicate). In the prototype tool under development, rules are implemented as PROLOG clauses. The enactment mechanism of the guided process is therefore, built on top of the PROLOG inference engine. More details about rules can be found in 21 .
3 Guiding the Description of the Electricity Application Fulfilment Business Process This section describes the guided process of use case specification with the EAF (Electricity Application Fulfilment) example. The presentation is organised in five steps which show the progressive transformation, completion and integration of input scenarios describing the process of applying for electricity into a use case specification. With regard to the levels presented in section 1, in this example, we start with scenarios describing <> and end with the description of <~system interactions >>. Due to space limitation, the <~system internal >>level is not tackled in this paper. Let us assume that the use case has been situated and its contextual information provided and stored in the use case structure (see in appendix the corresponding specification). The first step that we consider is therefore, the capture of the initial textual scenario which is intended to describe a normal course of actions.
3.1 The Initial Scenario Capture Let us assume that after having studied the guidelines, the use case author writes down his view of the normal course of interactions between a user who applies for electricity and the Electricity Company (EC) as follows: The customer requests for a connection to the network. The commercial employee asks his identification papers and the location of the house to be connected. If the customer is not writtenoff and the installation exists the commercial employee asks to the customer to sign the contract and to pay the deposit. If the meter exists, a technical employee performs the connection to the network, but before the commercial employee sends a service order for connection. Then, the customer is informed that the connection to the network is done. The f i n a l states are : The customer is connected to the network. EC has a contract for this
:ustomer. EC has the network extended with a meter connection.
The text is then scanned with the analysis rules and mapped onto elements of the use case specification using mapping rules.
347
3.2 Linguistic Analysis and Mapping on the Episode Structure As part of the linguistic approach sketched in section 2, analysis rules are used to identify the text slrncture, and to map the text onto instances of sentence and clause semantic patterns. For example the analysis rule AN1 aims at generating the action pattern from a clause having a subject and a verb expressed in the active form. AN1 : 17"NG, V: NG(SubjecOObject V (Main Verb)Action GENERA TE(Action(V)Agent: ? ; Object:NGJ)
(VG active)Actio n --->
Applying all necessary rules results in our example in the following set of instantiated patterns : Communication (request) Agent:'the customer' ; Object:'connection to the network' ; Source:'the customer' ; Destination: ? Communication (ask) Agent:'the commercial employee' ; Object:'his identification papers and the location of the house to be connected' ; Source:'the commercial employee' ; Destination: ? Constraint Condition : State Object:'the customer' ; State:'not written-off', StateObject:'installation' ; State:'exists' ; Constrained: Sequence Before : Communication (ask) Agent: 'commercial employee' ; Object:'to sign the contract', Source: 'commercial employee'; Destination: 'the customer' After : Communication (ask) Agent: 'commercial employee' ; Object:' to pay the deposit' ; Source: 'commercial employee'; Destination: 'the customer'; SequenceBefore : Constraint Condition : State Object:'the meter' ; State:'exists'; Constrained : Sequence Before : Communication (send) Agent: 'commercial employee'; Object:'a service order for connection' ; Source: 'commercial employee' ; Destination: ? ; After : Action (perform) Agent:'a technical employee' ; Object:'the connection to the network' After : Communication (inform) Agent: 7; Object: 'the connection to the network is done' ; Source : ?; Destination: 'the customer' Thefinal states are : State Object:'the customer' ; State:'connected to the network' Ownership Owner:'EC IS' ; Owned:'contract for this customer' Ownership Owner:'EC IS' ; Owned:'network extended with a meter connection' Fig. 4. Instantiated patterns The analysis of the initial scenario leads to the instantiation of seven action clause patterns within which six are communication action clause patterns. The instantiation of the action and communication action clause patterns from the input text provides values to the agent, source, object and destination cases. Question marks charactefise missing elements in the input text ; they will be used for completion. The analysis rules identify the agents of the scenario out of the agent, source and destination cases : 'the customer; "the commercial employee' and 'the technical employee'. Then, the object case identifies the resources used in each atomic action of the flow of actions : 'connection to the network', 'his identification papers and the location o f the house to be connected', 'to sign the contract and to pay the deposit', 'a service order for connection ', and 'the connection to the network is done'.
348
Moreover, the analysis of the initial scenario shows that the actions are related through two sequence sentence patterns and two constraint sentence patterns. Based on these pattern instances, mapping rules are automatically used to produce a natural language specification of the flow of actions of the initial scenario (consequently, it is a single course of actions). For sake of clarity, we use an indented presentation of the flow of actions, and we associate a unique identification number to each action. 1. The customer requests for a connection to the network. 2. The commercial employee asks his identification papers and the location of the house to be connected. 3. If the customer is not written-off and the installation exists 4. Then 5. The commercial employee asks to the customer to sign the contract 6. The commercial employee asks to the customer to pay the deposit. 7. If the meter exists 8. Then 9. The commercial employee sends a service order for connection. I 0. A technical employee performs the connection to the network. 11. The customer is informed that the connection to the network is done Final states : The customer is connected to the network. EC has a contact for this customer. EC has the network extended with a meter connection. Fig. 5. Flow of actions and final states after the mapping of the initial scenario Let us comment this mapping. First, based on the action and communication action clause patterns instantiated during the initial text analysis, atomic actions are identified through rules MA1, MA2 and MA3. ,VIA1 : Iz" V, A, O, S, D : Communication (It) Agent:A ; Object:O ; Source.'S ; Destination:D A (Unify (,4, S) v Unify (A, D)) ~ GENERATE(Atomic Action (Name : V, From Agent : S, To Agent : D, Parameter : 0)) MA2 : 17"V, A, _70 : Action (V) Agent:A ; Object:O A Agent (0) ~ GENERATE(Atomic Action (Name : V, From Agent : A, To Agent : (3)) MA3 : V V, A, -_7 O: Action (V) Agent:A ; Object:O A Agent (0) GENERA TE(Atomic Action (Name : V, From Agent : A, To Agent : A, Parameter : (9)) As stated in these rules, the atomic actions are analysed separately, even if they occur in the same sentence. Communication action pattern instances lead to the mapping o f an atomic action. Our purpose being not to rephrase the use case author scenario, the expression o f the atomic actions identified in the initial text is not modified. Based on the sequence pattems, atomic actions have been organised in the right sequence. For example, the sentence "a technical employee performs the connection to the network, but before the commercial employee sends a service order for connection" has been split into "(9) The commercial employee sends a service order for connection. (10) A technical employee performs the connection to the network". When no explicit sequencing of the actions is expressed, the ordering of the sentences in the initial scenario is respected. Flow conditions such as "if the customer is not written-off' or "if the meter exists", are identified from constraint patterns. Once identified, the alternative flows o f actions
349
are isolated, and the corresponding flow conditions are put in the order provided by the constraint pattern instances. For example the sentence "If the meter exists, a technical employee performs the connection to the network, but before the commercial employee sends a service order f o r connection" becomes "(7) I f the meter exists (8) Then (9) The commercial employee sends a service order f o r connection (10) A technical employee performs the connection to the network". 3.3 Clarification and Completion of Actions The linguistic analysis provides a baseline for linguistic based clarification and completion of the identified atomic actions. Both rules rely on situations identifying possible linguistic ambiguities in the expression of the atomic actions. The clarification rules are used to change the wording and to remove possible ambiguities. Even if the first guideline recommends to "avoid the use o f anaphoric references such as he, she, it, his and him", it is necessary to check systematically the text provided by the author. The grammatical analysis performed as a pre-requisite for analysis rules provides the information required for these checks. Clarification rule CL1 uses this information and proposes to replace anaphoric references by nOunS.
CL1 .. V A : (Action Agent.'A ; Object.'_ v Action Agent.'_ ," Object.'A v Communication Agent:_ ," Object.'_ ; Source:A ; Destination:_ v Communication Agent: _ ;Object." _ ," Source: _ ; Destination:A v State Object: _ ; State:A vOwnership Owner:A ; Owned: _ v Ownership O w n e r : ; Owned:A v Localisation Location:A ) ,~ Anaphoric Reference (,4) --~ ASK(e Clarify A by replacing this anaphoric reference by a noun ~)) Note : ~ _ >~is used to denote an anonymous variable which value is of no importance. The predicate 'Anaphoric Reference (A)' identifies if the term A includes a pronoun (he, his, him, etc.).
The use case author has been using the pronoun "his" in the second sentence of his scenario. The clarification rule suggests to "clarify his by replacing this anaphoric reference by a noun". Taking this suggestion into account, he/she now decides to modify the flow of action 2 and replace "his" by "the customer's" which is a more explicit resource name. The action (2) thus becomes "The commercial employee asks the customer's identification papers and the location o f the house to be connected". As explained in the previous section, the instantiated patterns may highlight missing parameters through question marks. Some of the completion rules (e.g. CO5) help avoiding this form of incompleteness. In the EAF example, several communication action pattern instances are in the situation of one or several missing elements. For example, as shown in the analysis section, analysing the atomic action "the customer is informed that the connection to the network is done" instantiates the pattern Communication (inform) Agent: ?; Object." 'the connection to the network is done' ; Source : ?," Destination: 'the customer' where the agent of the action and the source of the communicated information are missing.
350
C05 : V V, 3 0." Communication (V) Agent ." ? ; Object : O, Source : ? ; Destination : ? /Atomic Action (V,_) --~ ASK(a Complete .' V I>y ... (agent of the communicatiotO fi'om... I (source of the communieatiotO to... (destination of the eommunicatio, 0 )~)
I
Applying the completion rule CO5 displays the following template and asks the use case author to fill it in : "the customer is informed that the connection to the network is done by... (agent initiating the communication)from... (source o f the communication) ". Making use of the template, the use case author completes the sentence which becomes "the customer is informed by the commercial employee that the connection to the network is done" in which the commercial employee is the agent and the source. The systematic application of linguistic completion rules leads to the completion of four actions. The new version of the current flow of actions specification is shown in figure 6 where the supplementary elements are in bold, and the elements that have been clarified in bold and italic. 1. The customer requests the commercial employee for a connection to the network. 2. The commercial employee asks to the customer the customer's identification papers and the location of the house to be connected. 3. If the customer is not written-off and the installation exists 4. Then 5. The commercial employee asks to the customer to sign the contract. 6. The commercial employee asks to the customer to pay the deposit. 7. Ifthemeter exists 8. Then 9. The commercial employee sends to the technical employee a service order for connection. 10. A technical employee performs the connection to the network. 11. The customer is informed by the commercial employee that the connection to the network is done.
Final states : The customer is connected to the network. EC has a contact for this customer. EC has the network extended with a meter connection. Fig. 6. Flow of actions after linguistic clarification and completion
3.4 Completing Action Dependencies In the use case model, atomic actions are refined by several sub-types: service request, service provision, information provision, information request, internal action, etc. Based on this typology, we defined action dependency patterns which state dependencies between several types of atomic actions. The non respect of these dependencies is captured in the situations of the completion rules. Rule CO8, for example, states that the provision of a service S from an agent B to an agent A should be preceded in the flow of actions by the request of the service S from AtoB.
351
C 0 8 : I/ I,'1, S, A, B : (Atomic Action(Name : VI, From Agent : B, To Agent : A, Parameter : S) A (Type(VI) = 'Service I'rovision ')) A --,(3 V2 : Atomic Action (Name : I/2, From Agent : A, To Agent : B, Parameter : S) A (Type(V2) = 'Service Request ') A Follow(Vl, I/2)) --~ ASK(r162 Complete service provision V1 with the anterior action :... (service request S from A to t3) ~)
Similarly, any service request should be followed by a service provision, this also applies to information requests and provisions. These patterns are exploited in other completion rules which are similar to CO8. Standalone requests or provisions of services and information can thus be identified and completed if necessary with their counterpart. Another action dependency pattern establishes the dependency between an alternative action and the action of verification of the corresponding flow condition. The corresponding rule is triggered when this dependency pattern is not respected in the flow of actions allowing to associate a verification action to the condition. A systematic application of the associated rules on our flow of actions of figure 7 leads to the following specification, in which the new elements are emphasised in bold. 1. The customer requests the commercial employee for a connection to the network. 2. The commercial employee asks to the customer the customer's identification papers and the location of the house to be connected. 3. The customer gives to the commercial employee the customer's identification papers and the location of the house to be connected. 4. The commercial employee cheeks if the customer record is not written-off and the customer record exists and the installation exists. 5. If the customer record is not written-offand the customer record exists and the installation
exists 6. Then 7. The commercial employee asks to the customer to sign the contract. 8. The commercial employee asks to the customer to pay the deposit. 9. The customer gives to the commercial employee the signed contract 10. The customer gives to the commercial employee the money for deposit. 11. The commercial employee gives back to the customer the customer's identification papers. 12. The commercial employee checks if the meter exists.
13. If the meter exists 14. Then 15. The commercial employee sends to the technical employee a service order for connection. 16. A technical employee performs the connection to the network. 17. The technical employee informs the commercial employee that the connection is done. 18. The commercial employee sends to the customer a copy of the contract.
19. The customer is informed by the commercial employee that the connection to the network is done.
352
Final states: The customer is connected to the network. The customer has the customer's identification papers. The customer has a copy of the contract. EC has a contact for this customer. EC has the network extendedwith a meter connection. Fig. 7. Flow of actions after the action dependencies completion. Let us comment the incremental changes occurred between figures 6 and 7. As completion rules are based on types of actions, the use case author is first asked to provide atomic action typing. He/she thus identifies that : 9 the "request connection to the network" action is a service request *, 9 the "ask the customer's identification papers and the location of the house to be connected" action is an information request, 9 the "ask to sign the contract "action is an information request, *the "ask to pay the deposit" action is a service request, 9 the "send a service order for connection" action is a service request, 9 the "perform the connection to the network" is an internal action, 9 the "inform that the connection to the network is done" action is a service provision *. The "ask the customer's identification papers and the location of the house to be connected" information request action, does not have a corresponding information provision. At this point, the use case author thinks about describing the interactions between the customer and the actors of the EAF in a more detailed way. Thus, he/she inserts the following sentence in the flow of actions "The customer gives his identification papers and the location of the house to be connected". Using the linguistic analysis and mapping rules, the sentence is then converted into the atomic action 3 of figure 7. Indeed, the linguistic completion and clarification are also performed, as presented in previous section. This has led to replace "his" by "the customer's", and to complete with a destination : "to the commercial employee". In the same way, the "ask to sign the contract" information request action is completed by the use case author inserting the following sentence "The customer gives to the commercial employee the signed contract". The same applies to "ask to pay the deposit". Finally, the "send a service order for connection" service request action from the commercial employee to the technical employee is completed by the corresponding service provision action "The technical employee informs the commercial employee that the connection is done". The application of the completion rules leads also to acknowledge that some parts of the specification are complete with respect to action dependency patterns. For example, the "request connection to the network" action is a service request from the customer to the commercial employee with the corresponding provision of the requested service (connection to the network). This correspondence is described by the use case author (see the two * in the list of typed actions provided above) at the same time than the types of actions. Thus, the description is already complete with respect to the satisfaction of a requested service. Note also that the use case author may decide not to apply the suggested completion, There are two flow conditions in the specification : (a) "the customer is not written-off and the installation exists", (b) "the meter exists". They do not have a preceding
353
action for checking the flow condition. Asking the use case author to verify the need for two new actions for the condition checking leads to complete the episode specification with "(4) The commercial employee checks if the customer is not written-off and the installation exists" and (12) "The commercial employee checks if
the meter exists". Using the corresponding completion rule, the use case author is asked to verify that the if these two conditions are complete. This leads him to insert another condition in the fourth sentence giving thus "(4) The commercial employee checks if the customer is not written-
off and the customer exists and the installation exists". Final states have also to be completed, using the associated nile. The use case author is asked to verify their completeness and if needed provides the missing elements. With regard to the added elements, the use case author is asked to provide the necessary action for the final states to be reached. Similarly, the use case author is asked to verify that the conditions referring to agent names are either dealing with the agent itself or an abstract representation of the agent. For example, in action n ~ 4 of figure 7, the condition "the customer exists" may mean that either the customer is in the EC office or that he is known in the record of the company. This leads to replace "customer" by "customer record" giving thus" (4) The commercial
employee checks if the customer record is not written-off and the customer record exists and the installation exists". 3.5 Reasoning on Flow Conditions
A flow of actions issued from a scenario description involves several flow conditions. For instance, in the current version of the normal episode of the EAF use case, there are four flow conditions : - if the customer record is not written-off - if the customer record exists - if the installation exists - if the meter exists As a consequence of the scenario definition, this flow of actions together with the flow conditions constitute a pathway that permits the user to reach successfully one of the possible final states of the episode. In the case above, the flow of actions leads to an happy EC customer with the connection to the network. As presented in section 2, the normal episode may comprise several pathways, and be complemented by exceptional episodes. Each of them permits to reach one of the possible final states. We believe that reasoning on the flow conditions of the initial scenario description can help to discover exceptional episodes, and new pathways of the normal episode. The emergence rules based on flow conditions support this reasoning. These rules raise the questions of what happens if the flow conditions do not hold. Based on the flow conditions of a given pathway all combinations of negative conditions are calculated. Thanks to emergence rules, the use case author is first asked to characterise all of them being either normal, exceptional or meaningless. Second, if two combination of conditions lead to describe the same scenario, then it should be clarified by the use case author. The result of applying these rules leads to the foUowing table.
354
(l!
customer is written-off
: exceptional
(2) (cltstomer is not written-off) & (customer does not exisO & (installation : normal exists') & Oneter exists)
O) (customer is trot written-off) & (customer exists'.) & (installation exists) : normal & (meter does ,rot exis0 (customer is trot written-ofJ) & (customer does not exisO & (installation exists). & Oneter does not exis0 (5i (customer is not written-off) & (customer exists) & (installation does not exis0 & Oneter does not exis0 (6) (customer is not written-off) & (customer does not exisO & (installation i does not exisO & Oneter does trot exist) (7) l (customer is not written-off) & (customer exists) & (installation does trot exisO & (meter exists) (8) (customer is trot written-off) & (customer does trot exisO & (installation does trot exisO & Oneter exists)
(4)
: normal : normal : normal : meaningless : meaningless
For each normal and exceptional scenario, a new iteration in the specification process starts. This includes the activities 2, 3 and 4 presented in section 2.3 : the textual scenario description provided by the use case author, its analysis and completion and its integration in the use case specification. There are two kinds of integration : the integration of an exceptional episode, and the integration of a new course of actions in the normal episode. The integration of an exceptional episode is simple and consists in relating the exception to an action of the normal episode. The integration of a new normal course of actions in the normal episode requires to embed the new flow of actions in this episode. The guidelines proposed to the use case author during the capture of these new scenarios are the same as the ones used to capture the initial scenario. In addition, we offer a copy & paste facility which enables to duplicate in the scenario under writing, a course of actions which already exists in the specification. This facility is also used by the tool to provide to the author a partially written scenario description and ask him to complete it. The following text illustrates this step of the process for case (1). The use case author's writing is in bold whereas the text automatically generated by the guidance tool is in italics, the use of the copy and paste functionality is mentioned as a comment between the signs "/* "and " * / " /*copy & paste action(s) 1 to 2 of the normal episode */ Tire customer requests the commercial employee for a connection to the network. The conunercial employee asks' to the otstomer the customer ~ ulentification papers and tire location of the house to be connected. The customer gives to the commercial employee the customer's identification papers and the location of the house to be connected. The commercial employee checks if the customer record is not written-off and the customer record exists and the installation exists. If 7(the customer record is not written-of~ Then the commercial employee informs the customer that he is written-off and the connection to the network can not be done.
Final states : The customer is not connected to the network.
355
The NL text is analysed and completed in a similar way as the one illustrated for the initial scenario. The resulting exceptional episode specification is the following. Exceptional episode of the use case Electricity Application Fulfibnent Name : WrittenoffCustomer Occurrence condition : When " (the customer record is not written-off). Where : the action 5 of the NormalCase episode. Action : 1. The commercial employee informs the customer that the customer record is written-off and the connection to the network can not be done. 2. The commercial employee gives back to the customer the customer's identification papers.
Final states : The customer is not connected to the network. The customer has the customer's identificationpapers. Proceeding in the same way with the scenario number 2 leads the use case author to describe the flow o f actions when the customer does not exist, the installation exists and the meter exists as shown in figure 8.
/*copy & paste action(s) 1 to 4 o f the normal episode */ The customer requests the commercial employee for a connection to the network. I f the customer record is trot written-off Then If 7(the customer record exists) Then The commercial employee creates the customer record /*copy & paste actions 7 to 10 of the normal episode */ If(the installation exists) Then The commercial employee asks to the customer to sign the contract The commercial employee asks to the customer to pay the deposit. The customer gives to the commercial employee the signed contract. The customer gives to the commercial employee the money for deposit The commercial employee gives back to the customer the customer's identification papers. The commercial employee checks if the meter exists. If(the meter exists) Then /*copy & paste actions 13 to 17 of the normal episode */ The commercial employee semis to the technical employee a service order for connection. A technical employee performs the connection to the network. The technical employee informs the commercial employee that the meter connection is done. The commercial employee sends to the customer a copy of the contract. The customer is informed by the commercial employee that the connection to the network is done. Final states: The customer is connected to the network. The customer has the customer's identification papers. The customer has a copy of the contract. EC has a new customer. EC has a contact for this customer. EC has the network extended with a meter connection. Fig. 8.The scenario number 2 as provided by the use case author
356
Note that the copy & paste functionality is used in this case by the use case author to introduce actions 7, 8, 9, 10, 11, 12, 15, 16, 17, 18 and 19 o f the normal episode in the current scenario. The analysis and completion steps are performed to obtain a specification that can be integrated in the use case specification. The integration step stands in two parts : the integration of the final states and the integration of the actions. The integration of the final states leads to add the keyword "sometimes" when a final state exists in the normal episode but not in the new pathway and vice versa. This results in adding "sometimes" before the final state "EC has the network extended with a meter installation" in the normal episode during the integration of scenario number 5. Then the integration rules are applied to re-organise the flow of actions of the normal episode. The integration of all the normal flows of actions, namely cases (2), (3), (4), (5) and (6) leads to the normal episode of figure 9. An asterisk marks a flow condition related to an exceptional episode. The same reasoning can be recursively applied to the new flow conditions (33) and (39). This leads to the emergence and specification of two exceptional episodes, namely "NetworkConnectionAborted" and "InstallationOnly" that are described in the appendix. The appendix presents all exceptional episodes of the EAF use case. Normal episode name : NormalCase action : 1. 'he customer requests the commercial employee for a connection to the network. 2. The commercial employee asks to the customer the customer's identification papers and the location of the house to be connected. 3. The customer gives to the commercial employee the customer's identification papers and the location of the house to be connected.. * 4. The commercial employee checks if the customer record is not written off and the customer record exists and the installation exists 5. If the customer record is not written off* 6. Then 7. If(the customer record exists) 8. Then 9. The commercial employee creates the customer record 10. If (the installation exists) 11. Then 12. The commercial employee asks to the customer to sign the contract. 13. The commercial employee asks to the customer to pay the deposit. 14. The customer gives to the commercial employee the signed contract. 15. The customer gives to the commercial employee the money for deposit. * 16. The commercial employee gives back to the customer the customer's identification papers. 17. The commercial employee checks if the meter exists. 18. If(the meter exists) 19. Then 20. The commercial employee sends to the technical employee a service order for meter connection.
357
21. A technical employee performs the connection to the network. 22. The technical employee informs the commercial employee that the meter connection is done. 23. Else 24. The commercial employee sends to the technical employee a service order for meter installation and a service order for meter connection. 25. A technical employee performs the meter installation and the connection to the network. 26. The technical employee informs the commercial employee that the meter installation and the meter connection are done. 27. Else 28. The commercial employee requests to technical employee to investigate the site. 29. The technical employee performs investigation 30. The technical employee informs the commercial employee that the investigation is done. 31. The commercial employee calculates price 32. The commercial employee asks to the customer to pay for the installation. 33. If the customer pays the commercial employee for installation * 34. Then 35. The commercial employee sends to technical employee a service order for installation. 36. The technical employee performs installation 37. The technical employee informs the commercial employee that the installation is done. 38. The customer is notified by the commercial employee that the installation is done. 39. If the customer asks to the commercial employee a connection to the network * 40. Then 41. The commercial employee asks to the customer to sign the contract. 42. The commercial employee asks to the customer to pay the deposit. 43. The customer gives to the commercial employee the signed contract. 44. The customer gives to the commercial employee the money for deposit. * 45. The commercial employee sends to the technical employee a service order for meter installation and a service order for meter connection. 46. A technical employee performs the meter installation and the connection to the network. 47. The technical employee informs the commercial employee that the meter installation and the meter connection are done. 48. The commercial employee sends to the customer a copy of the contract. 49. The customer is informed by the commercial employee that the connection to the network is done.
358
Final states: The customer is connected to the network. The customer has the customer's
identification papers. The customer has a copy of the contract. Sometimes, EC has a new customer. EC has a contact for this customer. EC has the network extended with a meter connection. Sometimes, EC has the network extended with a meter installation. Sometimes, EC has the network extended with an installation. Fig. 9. Version 4 of the Normal episode 3.6 C o m p l e t i n g the Use Case Specification with <<S y s t e m Interactions >>
N o w that we have integrated all scenarios describing the interactions between the human agents within the use case specification, we shall concentrate on the interactions between the agents of the organization and an automated system that shall support the business process, what is called ~ system interactions >>in section 1, figure 1. To this end, we have defined a set of completion rules which aim at querying the use case author about the requirements for a computerized system that can support the performance of the process. The rules concentrate on the both the communication and internal actions. For each communication action, the author is asked if the communication is supported by the system. I f this is so, the author completes the action accordingly. Similarly, for each internal action, the use case author is asked if the action is supported in some way by the system. This also leads to the emergence of new requirements for the system internal and to the completion of the system interactions. Using these rules, the dialogue leads to complete the normal episode with system interactions. The result is shown in figure 10 (EC IS stands to the Electricity Company information system, it is the information system that supports the EAF business process). 1. The customer requests the commercial employee for a connectionto the network. 2. The commercial employee asks to the customer the customer's iden"tfficationpapers and the location of the house to be connected. 3. The customer gives to the commercial employee the customer's identification papers and the location of the house to be connected.. * 4. The commercial employee requests to EC IS if the customer record is not written-off and the customer record exists and the installation exists 5. EC IS checks if the customer record is not written-off and the customer record exists and the installation exists 6. EC IS informs the commercial employee if the customer record is not written-off and the customer record exists and the installation exists
7. If the customer record is not written-off* 8. Then 9. If (the customer record exists) 10. Then 11. The commercial employee requests to EC IS to create the customer record 12. EC IS creates customer record 13. EC IS acknowledges the creation of customer record to the commercial employee
14. If(the installation exists)
359
15. Then 16. The commercial employee asks to the customer to sign the contract. 17. The commercial employee asks to the customer to sign the contract to pay the deposit.* 18. The customer gives to the commercial employee the signed contract. 19. The customer gives to the commercial employee the money for deposit.* 20. The commercial employee gives back to the customer the customer's identification papers. 21. The commercial employee requests to EC IS if the meter exists. 22. EC IS checks tithe meter exists 23. EC IS informs the commercial employee if the meter exists 24. If(the meter exists) 25. Then 26. The commercial employee requests to EC IS to send to the technical employee a service order for meter connection. 27. EC IS sends to the technical employee a service order for meter connection 28. A technical employee performs the connection to the network. 29. The technical employee informs the EC IS that the meter connection is done. 30. EC IS informs the commercial employee that the meter connection is done 31. Else Fig. 10. Version 5 of the normal episode completed with system interactions 4. Conclusion Activities, such as business process engineering, business process re-engineering or business process improvement call for accurate description o f business processes. Our proposal is about an approach supporting the construction of BP specifications. A BP specification takes the form of a use case comprising information about the context of the BP, the interactions between the agents involved in the BP, the interactions of these agents with a computerised system supporting the BP and a set of internal system requirements. We propose to guide the construction of a use case specification for BP using textual scenarios. A scenario describes, in natural language, interactions between different agents (i.e. a service requester and some service providers) and internal actions involving a single agent. An interaction is expressed as a commtmication action between two agents. A use case specification integrates all normal and exceptional scenarios describing a BP. We use the case grammar presented in 21 to analyse and to extract the semantics of textual scenarios. On top of the linguistic approach, the construction of use case specification is guided. Guidance is based on use case model knowledge and takes the form of rules which encapsulate knowledge about the use case model concepts in order to facilitate (1) the completion of scenarios, .(2) emergence of other normal or exceptional scenarios and (3) integration of scenarios into a complete use case
360
specification. The guided process was illustrated with a real case study dealing with the business process "Electricity Application Fulfilment" borrowed to an Electricity Company. In this paper, we focused on the interactions between the organisational agents of the studied business process, and their interactions with the automated system supporting the BP. Our current work consists in extending the guidance rules to support the emergence of system internal requirements on one hand, and system contextual requirements on the other hand. The former relates to internal system objects and behaviour whereas the latter deals with the context in which the business process takes place (weaknesses, opportunities, non functional requirements, etc,). Meanwhile, we are completing the current PROLOG implementation to handle the entire set of rules presented in this paper.
References 1. J.M. Caroll, The Scenario Perspective on System Development, in J.M. Carroll (ed.), Scenario-Based Design: Envisioning Work and Technology in System Development (1995). 2. A. Cockburn, Structuring use cases with goals, Technical report, Human and Technology, 7691 Dell Rd, Salt Lake City, UT 8 4 1 2 1 , HaT.TR.95.1, http://members.aol.com/acocburn/papers/usecases.htm (1995). 3. B. Dano, H. Briand, F. Barbier, A Use Case Driven Requirements Engineering Process, In Third IEEE International Symposium On Requirements Engineering (RE'97), Antapolis, Maryland (IEEE Computer Society Press, 1997). 4. E. Dubois, P. Heymans, M. Jarke, K. Pho, K. Weidenhaupt, A. Suteliffe and N.A.M. Maiden, Integration of Scenario Representations: Formalisation and Examples, ESPRIT Reactive Long Term Research Project 21.903 CREWS, Deliverable W4: Knowledge Representation Working Group (1997). 5. T. Erickson, Notes on Design Practice: Stories and Prototypes as Catalysts for Communication, in John M. Carroll (ec), Scenario-Based Design: Envisioning Work and Technology in System Development (John Wiley and Sons, 1995) 37-59. 6. C. Fillmore, The Case For Case, in Holt, Rinehart and Winston (eds.), Universals in Linguistic Theory (Bach & Harms, 1968) 1-90. 7. J.D. Gould, How to design usable systems, in M. Helander (ed.), Handbook of HumanComputer Interaction (Elsevier Science, 1988) 757-790. 8. Hammer M. Champy J., "Re-engineering the corporation : a manifesto for business revolution", Harper Collins publishers, inc., New York, 1993. 9. L Jacobson, The Use Case Construct in Object-Oriented Software Engineering, in John M. Carroll (ed.), Scenario-BasedDesign: Envisioning Work and Technology in System Development (John Wiley and Sons, 1995) 309-336. 10. I. Jacobson, M. Ericsson and A. Jacobson, The Object Advantage, Business Process Reengineering with Object Technology (Addison-Wesley Publishing Company, 1995). 11. M. Jarke, K. Pohl, P. Haumer, K. Weidenhaupt, E. Dubois, P. Heymans, C. Rolland, C. Ben Achour, C. Cauvet, J. Ralyte, A. Sutcliffe, N.A.M. Maiden and S. Monicha, Scenario Use in European Software Organisations - Results from Site Visits and Questionnaires, ESPRIT Reactive Long Term Research Project 21.903 CREWS, Deliverable WI: Industrial problem capture Working Group (1997). 12. J. Karat, Scenario Use in the Design of a Speech Recognition System, in John M. Carroll (ed.), Scenario-Based Design: Envisioning Work and Technology in System Development (John Wiley and Sons, 1995) 109-135.
361
13. K. Koskimies and H. Mossenbock, Scene: Using Scenario Diagrams and Active Text for illustrating Object-Oriented Programs, in Proc. oflCSE-18 (1995) 366-375. 14. M. Kyng, Creating Contexts for Design, in John M. Carroll (ed.), Scenario-Based Design: Envisioning Work and Technology in System Development (John Wiley and Sons, 1995) 85-107. 15. R.L. Mack, Discussion: Scenarios as Engines of Design, in John M. Carroll (ed.), Scenario-Based Design: Envisioning Work and Technology in System Development (John Wiley and Sons, 1995) 361-387. 16. J. Nielsen, Scenarios in Discount Usability Engineering, in John M. Carroll (ed.), ScenarioBased Design: Envisioning Work and Technology in System Development (John Wiley and Sons, 1995) 59-85. 17. C. Ports, K. Takahashi and A.I. Anton, Inquiry-based Requirements Analysis, in IEEE Software 11(2) (1994) 21-32. 18. J.C.S. do Prado Leite, G. Rossi, F. Balaguer, A. Maiorana, G. Kaplan, G. Hadad and A. Oliveros, Enhancing a requirements baseline with scenarios, In Third IEEE International Symposium On Requirements Engineering (RE'97), Antapolis, Maryland (IEEE Computer Society Press, 1997) 44-53. 19. B. Regnell, K. Kimbler and A. Wesslen, Improving the Use Case Driven Approach to Requirements Engineering, in the Second IEEE International Symposium OnRequirements Engineering, York, England (I.C,S. Press, March 1995) 40-47. 20. S.P. Robertson, Generating Object-Oriented Design Representations via Scenarios Queries, in John M. Carroll (ed.), Scenario-Based Design: Envisioning Work and Technology in System Development (John Wiley and Sons, 1995) 279-308. 21. C. Rolland, C. Ben Achour : Guiding use case specification in natural language, to appear in the Data and Knowledge Engineering Jottrnal, 1998. 22. M.B. Rosson and J.M. Carroll, Narrowing the Specification-Implementation Gap, in John M. Carroll (ed.), Scenario-BasedDesign: Envisioning Workand Technology in System Development (John Wiley and Sons, 1995), 247-278. 23. J. Rumbaugh, M. Blaha, W. Premerlani, F. Eddy, and W. Lorensen, Object-Oriented Modeling and Design, (Prentice Hall, 1991). 24. J. Rumbaugh and G. Booth, Unified Method, Notation Summary Version 0.8 (Rational Software Corporation, 1996). 25. K. Weidenhaupt, K. Pohl, M. Jarke, P. Haumer, Scenario Usage in System Development : a Report on Current Practice, ESPRIT Reactive Long Term Research Project 21.903 CREWS, Deliverable W 1-D2, (1997). 26. J. Whiteside, J. Bennett and K. Holtzbtatt, Usability Engineering : Our experience and evolution, in M. Helander (ed.), Handbook of Human-Computer Interaction (Elsevier Science, Amsterdam, 1988) 791-818. 27. R. Wirfs-Brock, Designing Objects and their Interactions: A Brief Look at Responsibilitydriven Design, in John M. Carroll (ed.), Scenario-Based Design: Envisioning Work and Technology in System Development (John Wiley and Sons, 1995) 337-360.
362
Appendix
T h e use case specification is c o m p o s e d o f the contextual information s h o w n below, one normal episode s h o w n in figure 9 and four exceptional episodes. C o n t e x t u a l information
Use case name Initiating agent Goal Initial states
Electricity Application Fulfilment A customer o f the Electricity C o m p a n y (EC) ' ' To connect the customer to the c o m p a n y network Customer is present in the EC office with a completed application form.
Exceptional episode of the use case Electricity Application Fulfilment Name : WrittenoffCustomer Occurrence condition : When 7 (the customer record is not written-off). Where : the action 5 of the NormalCase episode. Action :
1. The commercial employee informs the customer that the customer record is written-off and the connection to the network can not be done. 2. The commercial employee gives back to the customer the customer's identification papers. Final states : The customer is not connected to the network. The customer has the customer's
identification papers.
Exceptional episode of the use case Electricity Application Fulfilment name : NoIdentifieationPaper occurrence condition : When q (the customer gives to the corram~ial employee the customer's identification
papers and the location of the house to be connected ). Where : the action 3 of the NormalCase episode. action :
1. The commercial employee informs the customer that the connection to the network can not be done without customers identification paper's. Final states : The customer is not connected to the network.
Exceptional episode of the use ease Electricity Application Fulfilment name : NetworkConnectionAborted occurrence condition : When ~ (the customer gives to the conmaercial employee the signed contract and the money for deposit) OR 7(the customer pays the commercial employee for installation), Where : the action 15 and a year after action 32 of the NormalCase episode. action : 1. The commercial employee informs the customer that the connection to the network can not be done without payment. Final states : The customer is not connected to the network. The customer has the customer's identification papers. Exceptional episode of the use case Electricity Application Fulfilment name : InstallationOnly occurrence condition : When ~(the customer asks to the c o ~ i a l employee a connection to the network) OR
(the customer gives to the commercial employee the signed conlract and the money for deposit). Where : the action 43 and 6 months after action 38 of the NormalCase episode. action :
1.
The commercial employee informs the customer that the network connection request is aborted.
Final states : The customer is not connected to the network. The customer has the customer's
identification papers., EC has the network extended with an installation.
Building Quality into Case-Based Reasoning Systems Igor Jurisica 1 and Brian A. Nixon 2 1 University of Toronto, Faculty of Information Studies 140 St. George St., Toronto, ON M5S 3G6, Canada [email protected] 2 University of Toronto, Dept. of Computer Science Toronto, ON M5S 3H5, Canada [email protected] A b s t r a c t . Complex decision-support information systems for diverse domains need advanced facifities, such as knowledge repositories, reasoning systems, and modeling for processing interrelated information. System development must satisfy flmctional requirements, but must also systematically meet global quality factors, such as performance, confidentiality and accuracy, called non-functional requirements (NFRs). To build quality into an important class of decision support systems, case-based reasoning (CBR) systems, this paper presents "QualityCBR," a goal-oriented, knowledge-based approach for systematically dealing with NFRs for CBR systems. With the idea that similar problems have similar solutions, CBR systems store cases (problems with solutions) and solve new problems by retrieving and reusing similar past cases. QualityCBR integrates existing work on CBR and NFRs. It helps developers state and refine NFRs, consider tradeoffs, make decisions and evaluate their impact on NFRs. We illustrate the approach in a complex medical domain, in vitro fertilization, where CBR suggests therapy for patients, predicts the probability for successful pregnancy, and determines patient's characteristics that can improve pregnancy rate.
1
Introduction
Complex information systems in both the public and private sectors need a number of advanced facilities, including decision support systems (DSSs), repositories, reasoning systems, and facilities for modeling and processing large amounts of complex information. Many DSSs have been built for individual domains in an ad hoc manner. However, to effectively build families of DSSs for complex domains, such as medical, governmental or industrial applications, we need a systematic knowledge-based approach to: (1) empower expert user to make effective decisions using a DSS, and (2) address concerns for quality and performance requirements. Case-based reasoning (CBR) systems 25 are an important class of DSSs that represent experiences (problems with solutions) as cases. Cases are used for solving new problems by accessing past cases and comparing their similarity to a given problem. In this paper we use a generic CBR system called Tr
364
(pronounced tah-tree) to build a complex medical DSS, which can be used to advise physicians who prescribe treatment plans for in vitro fertilization (IVF) patients 24. CBR systems must meet functional requirements, including retrieving past cases, selecting and reasoning about relevant ones, interactively exploring cases, and adapting them to produce a solution, which is then evaluated. In addition, CBR and other large and complex information systems must meet non-functional requirements (NFRs or quality requirements), which are globM requirements for quality factors such as performance, accuracy and confidentiality. NFRs are important for the success of complex systems. In the case of medical systems, confidentiality is crucial. Dealing with NFRs systematically is difficult, because a developer must consider not only requirements, but also implementation alternatives and tradeoffs. In addition, requirements can be in conflict with each other (e.g., it may not be possible to have both expressive representation and fast access time). There are many implementation alternatives, with impact on different NFRs (e.g., having entry clerks type information twice might improve accuracy, but decrease user friendliness). Decisions are interrelated, with a global impact on the target system. For these reasons, one can't simply use "canned alternatives" to meet the quality requirements. Instead, we use an approach where the developer considers the characteristics of a particular system being developed and application needs in a systematic way. This provides a process that helps producing customized systems that meet quality requirements. Simple applications can usually be built in an ad hoc manner, and dealing with requirements may not be difficult. However, a distinguishing aspect of large and complex information systems, whether medical, governmental or industrial, is that characteristics including data, algorithms, domains, requirements, priorities and workload must all be considered. Furthermore, these characteristics interact in complex ways. Hence it is important to deal with them in a systematic way. We deal with the complexity of these kinds of systems by: using a knowledgebased approach which catalogues expertise, offering competent and efficient CBR facilities, and using a structured approach to deal with NFRs. These facilities are combined in our approach, called QualityCBR. To provide a development process that addresses NFRs for CBR, and is goaloriented, systematic developer-directed and qualitative, we draw on the "NFR Framework" 4, 6, 31. The NFR Framework supports this process of building quality in to a system interactively, while considering NFRs throughout the development process. Quality requirements are treated as goals to be systematically achieved during the design and development process. The NFR Framework, with its associated tool, helps a developer state and refine NFRs, consider design tradeoffs, justify decisions and evaluate their impact on NFRs, while giving the developer control of the development process. To deal with performance requirements, we draw on the Performance Requirements Framework 33, 34, 35, a specialization of the NFR Framework.
365
The factors which must be considered during development may all change during a system's lifetime. This greatly increases the complexity of development, and further motivates the need for a systematic approach. By using a knowledgebased approach, and by drawing on the NFR Framework's facilities for dealing with change 7, we can systematically deal with change. There are two possible combinations of techniques for CBR and NFRs: (1) using CBR to support reasoning about and reuse of NFRs, and (2) using NFRs to systematically build quality into a CBR system. This paper addresses the latter issue, using the QualityCBR approach. In particular, we describe the process of using QualityCBR and providing catalogues that deal with alternative implementations. QualityCBR draws on a flexible knowledge representation language for information systems - - Telos 30, relevance assessment 20, similarity-based retrieval algorithm 22, and the NFR Framework's goal-oriented, qualitative approach 31. In addition, QualityCBR uses knowledge discovery algorithms 1, 24 and data model implementation experience 36. QualityCBR is applied to a complex medical DSS for IVF practitioners T.A3 24. During the development of the system we considered some NFRs, albeit in an ad hoc manner. We show how a developer could use QualityCBR to systematically build a CBR system for IVF by addressing NFRs such as performance - "Select relevant cases quickly", confidentiality - "Store patient records securely", and informativeness - "Display results informatively". We also consider the impact of some changes.
2
The
QualityCBR
Approach
This section presents the elements of the QualityCBR approach, for addressing non-functional requirements of case-based reasoning systems. Traditionally, CBR system were developed for a specific application. The presented work aims at defining a generic framework that is adaptable for different domains, while ensuring that both functional and non-functional requirements are systematically met. 2.1
Case-Based Reasoning
This section describes principles of case-based reasoning (CBR) and a particular prototype T.A3. Our aim is a flexible system that can be applied to various domains, without sacrificing system performance. We consider system performance as a quality of solution and its timeliness. A case-based reasoning approach 25 relies on the idea that similar problems have similar solutions. Facing a new problem, a CBR system retrieves similar cases stored in a case base and adapts them to fit the problem at hand. Informally, a case comprises an input (the problem), an output (the solution) and feedback (an evaluation of the solution). CBR involves the process of: (1) Accepting a new problem description; (2) Retrieving relevant cases from a case base (past problems with similar input); (3) Adapting retrieved cases to fit the input
366
problem and finding a solution to it; and (4) Evaluating the solution (producing feedback for the case). Considering the above CBR cycle, one can say that the more similar the eases are, the less adaptation is necessary, and consequently, the proposed solution may be more correct. Then, an important task is how to measure case relevance (similarity or closeness) to guarantee retrieving only highly relevant cases, i.e., cases that are similar according to specified criteria, and thus can be useful in solving the input problem in a particular context. Thus, we need a variable-context similarity assessment. In many processes, it is better to retrieve fewer cases, or none, than to retrieve less useful eases that would result in a poor solution. But similarity of cases is only one measure of system quality. It is also important that the solution be provided quickly. It should be noted that the tradeoff between closeness and timeliness of a solution depends on requirements of a particular application 19. For these reasons we use a variable-context similarity assessment and case base clustering as described next.
TA3 is a CBR system, which uses a variable-context similarity-based retrieval algorithm 22 and a flexible representation language. Knowledge must be represented in a form appropriate for the intended user, and the representation should be rich enough to support complex, yet efficient processing 23. Cases are represented as a collection of attribute-value pairs. Individual attributes are grouped into one or more categories 22. Categories bring additional structure to a case representation. This reduces the impact of irrelevant attributes on system performance by selectively using individual categories during matching. As a result, we get a more flexible reasoning system 19, a more comprehensible presentation of complex information 20, improved solution quality 24, and improved scalability 23. During the CBR process, we want to handle partial as well as exact matches. We have a partial matchin9 when attribute values of one case match only a subset of values of another case. In order to retrieve and control both exact and partial matching, a view of a case, called a context, is defined. Thus, a case to be interpreted in a given context. By controlling what constitutes a partial match, context specifies important attributes and how "close" an attribute value must be. We say that a case satisfies (or matches) a particular context, if for each attribute specified in the context, the value of that attribute in the case satisfies the constraint 22. Thus, the matching process can be described as a constraint satisfaction problem 40. The quality of the matching process is measured by the closeness of retrieved cases 22, timeliness of the answer 23, and wzlaptability of the suggested solution 26. Ortega has shown that partial m-of-n matches improve performance if rn is reasonably selected 37. Our approach of representing cases as sets of Telosstyle categories 30, each comprising a set of tuples, allows for multiple levels of m-of-n matching. Thus, important attributes may require n-of-n matches for a given category, and less important attributes may allow for k-of-n matches (k < n). The problem is to find these attribute groupings, i.e., a context that specifies which attributes are needed for accurate prediction, and what range or similarity should be allowed for attribute value.
367
This knowledge can be automatically discovered 24 and can be used for case base clustering by: (1) appropriately grouping attributes into categories (clustering of attributes); (2) discovering what values are "close" for particular attributes (clustering of attribute values); and (3) structuring the case base into clusters of relevant cases (clustering of cases).
2.2
Handling Non-Functional Requirements
The NFR Framework 4, 31 helps a developer represent and use key concepts about NFRs (e.g., security and performance), the particular domain (e.g., IVF), and development expertise, (e.g., CBR, databases and system development). Being influenced by work in DSSs 28, the NFR Framework maintains a concise and structured development graph whose components record the developer's goals, decisions and design rationale. The developer states a set of NFRs for the system, which are represented as goals that are considered throughout the system development process. In trying to meet the requirements, developers are helped in choosing among design alternatives, which are organized in a catalogue of methods. Partial positive or negative relationships among goals are recorded as qualitative link types. Knowledge of design tradeoffs is arranged in a catalogue of correlation rules. After decisions are made, the NFR Framework uses its evaluation algorithm to help the developer determine if overall goMs have been met. Section 3.2 presents the components of the NFR Framework in more detail, and illustrates their use. The NFR Framework has been previously applied to information systems in several domains, in both the public and private sectors (e.g., health insurance, banking and government systems) 5, 7. Its approach can be specialized to deal with a number of NFRs, such as performance 33, 34, 35, accuracy 4 and security 3. For performance, for example, we represented principles for building good response time into systems 39 and arranged information system implementation knowledge using a layering approach 17 based on data model features, to reduce the number of issues considered at a time. The "NFR Assistant" prototype tool 4, provides support to a developer using the NFR Framework, by providing catalogues of concepts and methods, aiding the construction and evaluation of development graphs, and keeping track of correlations. The tool draws on the ConceptBase system 18 which uses the Telos 15, 30 knowledge representation language. A specialization of the tool, the Performance Requirements Assistant 34, 35, offers catalogues of concepts and techniques for treating performance requirements, using other Telos-based knowledge base management tools, 1 but offers only a subset of the functionality of the NFR Assistant. 1 M. Stanley's Telos sh and B. Kramer's RepBrowser, at the University of Toronto.
368
2.3
Cataloguing CBR and NFR
Knowledge
The QualityCBR approach organizes knowledge about issues and techniques for CBR and NFRs. These knowledge bases, represented in Telos, serve as a basis for recording experts' knowledge and are used during system development, They help a user to satisfy NFRs (such as performance and confidentiality), effectively use CBR techniques (e.g., knowledge representation, retrieval), and consider particular characteristics of the system under development (e.g., workload, confidentiality concerns).
Clustering Partial DKforK
D
Full ~ Domain Knowledge Explanation Database Data Based Knowledge Discovery Learning Techniques (DK) (KDD) (EBL) (DB)
Fig. 1. A Catalogue of Clustering Techniques for CBR. Some steps, typically done early in using the approach, involve defining and organizing a variety of types of knowledge applicable to the system under development. This produces a number of catalogues of concepts: - Concepts about a particular class of reasoning systems (e.g., CBR), such as components of the CBR system, problem decomposition techniques and implementation alternatives. Figure 1 shows a sample catalogue for implementation alternatives for clustering techniques. Specialized catalogues draw on combinations of aspects, e.g., domain knowledge for knowledge data discovery. - Concepts about particular NFRs (e.g., performance and security). For example, a terminology of performance concepts is made available, along with a catalogue which shows the impact of implementation techniques on time and space goals 34. - Concepts about the particular application domain, e.g., IVF: descriptions of processes (e.g., a cycle of patient treatment) and workload (e.g., number of patients). - Generic concepts associated with the NFR Framework, e.g., definitions of the components of development graphs which record developers' decisions.
369
3
Illustrating
the QualityCBR
Approach
This section shows the use of QualityCBR's components and cataloguing to support several NFRs for the IVF domain. Section 3.1 presents the domain of our study of the IVF system. We consider top level non-functional requirements for an IVF system, which could be stated by a developer, involving: performance patient records must be retrieved and analyzed quickly (Section 3.2), and confidentiality- records must be stored securely (Section 3.3). In addition, the system should be robust and user-friendly (Section 3.4). 3.1
F u n c t i o n a l R e q u i r e m e n t s in t h e I V F D o m a i n
In vitro fertilization (IVF) is an example of a complex medical domain, where DSS can be used to suggest the hormonal treatment and to support research 24. Individual patients respond to the treatment differently. A patient's response and the pregnancy rate depends on many attributes. While experienced doctors can use their knowledge to suggest a treatment for a patient, it is difficult for them to perceive trends and make informed decisions to optimize success rates for each individual infertile couple. This is especially a concern when knowledge about influencing factors changes. Pre&ction of the likelihood of pregnancy involves suggestion of a treatment. This is performed in two stages. First, given initial information about the patient (diagnosis, previous treatment history, etc.) the task is to find similar patients from the case base and make a suggestion of how to treat the current patient to increase the probability of successful pregnancy. This includes finding all relevant cases, and considering retrieved cases with pregnancy as successful examples and retrieved cases without pregnancy as negative cases. An adaptation process uses this information to suggest values for remaining attributes in the current case, namely how long the patient should be stimulated and what amount of the hormones should be used. Second, after the initial treatment is completed, additional attributes are available (patient's responsiveness to the hormonal stimulation). The task is then to predict the outcome of the whole treatment, i.e., to predict likelihood values for pregnancy and for unsuccessful cases. The prediction task can also be considered as an optimization problem: for a given patient minimize the amount of hormonal therapy required, without compromising the outcome. Knowledge discovery is used to find regularities in the case base by using knowledge-mining techniques, as well as to suggest missing data. Here, physicians have no particular case in mind, however, they may consider the whole knowledge base or only certain cases. Knowledge mining in 7,43 involves finding a context in which a particular group of cases is considered similar. The user has the ability to specify a threshold, which controls the quality and the quantity of discovered information 24. Considering that each patient is described by about a hundred attributes 24, that there are about 600 patients per year per clinic and that there are about 300 IVF clinics in North America 29, the problem is not simple. Moreover, IVF information is more sensitive than general medical information and the
370
complex IVF process involves various professionals, which need to access part or whole information about the patient. IVF has relevance to both the pubhc and private sectors. In the Province of Ontario, Canada, for example, publiclyfunded health insurance covers the cost of IVF for certain forms of infertility, e.g., tubal blockage, while others are not covered, and are handled by private clinics.
3.2
Dealing with P e r f o r m a n c e R e q u i r e m e n t s
We now show how performance requirements for the IVF domain are handled using QualityCBR. We also describe components of the NFR Framework used in QualityCBR. System performance is an important factor for complex applications. Good performance includes fast response time and low space requirements. For the IVF system, a developer might state that one important goal is to have fast response time when accessing patient records, for reasoning as well as case updating. This requirement is represented as a goal: Time P a t i e n t Records and Reasoning, as shown in Figure 2. Time is the sort of the goal (i.e., the particular NFR concept, addressed by the goal) and P a t i e n t Records and Reasoning is the parameter (i.e., the subject) of the goal. (The entries within circles will be discussed below.) Another main goal is to have fast response time for reasoning operations done by researchers, represented by Time Research Reasoning.
/
/ ~
Cla_'nn_"Aid Doctor"
~pdate
~Predietion ~
- - _
Time Research Reasoning
Time
Patient Records and Reasoning
~
~
,~--', Argument
\
~'~">~a.~./
/-"x ~ . ~ Satifficed Goal
~
\+
~
Legend NFRGoal
Discovery
DeniedGoal
9
I _ --~ Corrdation link t
J
+
Positive Link
I
j
"
VeryNegativeLink
, . . . . . . . . . . .
,. . . . . . . . . . . .
No Partial Full Clustering Clustering Clustering
Fig. 2. Dealing with Performance Requirements for Reasoning.
Using methods and catalogues of knowledge (for performance, CBR, IVF, etc.), goals can be refined into more specialized goals. Here, the developer used knowledge of the IVF domain to refine the time goal for patient information into two goals, one for good response time for updating patient records and the other for good response time for the retrieval and decision making process. These two offspring goals are connected by an And link to the parent goal. This means that
371
if both the goal for fast updates and the goal for fast prediction are accomplished then we can say that the parent goal of fast access to patient records will in some sense be accomplished. The NFR Framework takes a qualitative, "satisficing" approach, in which goals are more-or-less met, although they may not be satisfied in an absolute sense 38. Similarly, the goal of good response time for research reasoning can be refined into a goal of fast response for the "discovery" process which searches patient records for patterns. Here, the parent has one offspring, connected by a positive ("§ link, which indicates that accomplishing the offspring will contribute positively towards accomplishing the parent goal. Other types of relationships can be shown by other link types (see Figure 2). In building quality into a system, it is important to identify priorities. For the case of building performance in, we should identify time-critical operations as well as those which dominate the workload 39. Here, we identify the prediction operation as being time-critical (indicated by "!"), and provide a reason using domain knowledge: it is important to aid the doctor by quickly suggesting a treatment. This is an example of recording design rationale 28 the reasons for decisions - using the NFR Framework's arguments. As part of the development graph, arguments are available when making further decisions and changes. It is important to note that the developers use their expertise to determine what to refine, how to refine it, to what extent to refine it, as well as when to refine it. The NFR Framework and its associated tool help the developer, do some consistency checking, and keep track of decisions, but it is the developer who is in control of development process.
Implementation Alternatives. In moving towards a target system, one must consider implementation alternatives for case base clustering, which appropriately groups attributes, their values, and relevant cases together. The main concern for clustering is with the storage of patient records, which besides general patient information (name, address, etc.) consist of attribute-value pairs describing the diagnosis of infertility, previous and current treatments, the result, etc. Effective storage of this information facilitates the various CBR operations, because individual pieces of information have different importance and different effects on the treatment and on the overall outcome. Currently, the information is recorded in a paper-based form with general patient information being sent to a central hospital computer. A computerized IVF case base is populated in a batch process. Many of the implementation alternatives (shown as dark circles in Figure 2) will be drawn from appropriate catalogues. Implementation alternatives for the following clustering operations must be considered:
- Storage and update. In the IVF application, data entry and updates have the form of filling in blanks, either selecting a value from a pick-lists or typing it. Considering the amount of data in one clinic, storage and update are not major problems. However, taking into account possible extensions, e.g.,
372
linking several IVF clinics in a network to share their case bases, it is useful to note this requirement. - Prediction. A doctor uses the system to suggest a hormonal therapy for the current patient (see Section 3.1). It is important that the accuracy of predicted information is within reasonable bounds and a solution is provided swiftly. There is a relationship between accuracy, time and space: the more cases are stored, the more accurate solutions can be provided, but the longer it takes to find cases relevant to a given problem. - Knowledge discovery. Treatment protocols can be improved by using knowledge discovery 24. Discovered knowledge is used to organize attributes into categories, and cases into clusters (equivalence classes). The above considerations affect implementation alternatives ("satisficing goals") for case base clustering: (1) the system may not use any clustering; (2) it may use full clustering; or (3) an hybrid, a partial clustering scheme can be deployed; further variations of clustering from the methods catalogue can be considered (see Figure 1). Without clustering, updates are faster, as data need not be reorganized; however, prediction is slower as there is no clustering to aid the retrieval process. Thus, at the bottom left of Figure 2, No Clustering is shown to have a positive impact on update time, and a negative impact on prediction time. Full clustering is done by knowledge discovery: it speeds up prediction, but hinders update time. No and full clustering each slow down at least one of the three operations. The developer can formulate alternatives which reduce or avoid this problem. Partial clustering may start with cases clustered using domain knowledge, but may subdivide certain clusters into more detailed groups. Its main advantage is that it speeds up all three operations, instead of slowing any of them. However, no clustering is better ("++") than partial for update, and full clustering is better than partial for retrieval. Thus, partial clustering offers intermediate performance for some operations, but avoids bad performance for all of them. As a result, partial clustering is selected ("v ~') over the unchosen ("x") alternatives. Note, that an IVF facility that does not support research may give low priority to performance for knowledge discovery. Since the hormonal therapy suggestion would have high priority, full clustering would be selected.
Evaluating Goal Accomplishment. After decisions are made, the developer can determine their overall impact on the system's requirements. The developer is aided by the NFR Framework's semi-automatic evaluation algorithm, which examines the development graph, generally bottom-up. It starts with implementation decisions to accept ("x/~') or reject ("x") alternatives (shown in dark circles at the bottom of Figure 2), Results then propagate upward along evaluation links. Evaluation assigns values (e.g., "x/" or "x") to parent goals based on the values of offspring goals, and and the relationships (link types, e.g., "+" or %") between offspring and parent goals. For example, with a "+" link type, meeting ("~2') the offspring (e.g., Partial Clustering) helps meet ("x/") the parent; however, if the offspring
373
is denied ("• not achieved), the parent will be denied ("• The "-" link type can be thought of as giving the parent the "opposite" of the offspring's label. Values from all applicable offspring propagate to a parent. Here, partial clustering helps quickly accomplish updating, presentation and discovery. During the process, the developer may step in, to determine propagated values. For example, if a parent goal received positive and negative values from different offsprings, the developer is able to resolve the conflict using domain knowledge. It should be noted that not all goals can always be met, but performance can be enhanced if the priorities are accomplished 39. As presented in Figure 2, the critical goal for prediction has been met. Since the update time goal was also met, the top goal for records and reasoning was met. As the discovery goal was met, the top goal for research reasoning was also met.
Dealing with Changes in Priorities. Let's consider four imaginary IVF clinics with different priorities: (1)fast update of records, (2) fast prediction, (3) both fast prediction and fast update are important,~and (4)fast case base analysis (discovery). Depending on the priorities, we may adjust the solution of Figure 2 by choosing a different alternative. As a result, the first clinic would not use clustering, the second would use full clustering, and the third and fourth clinics would achieve their requirements by deploying partial (hybrid) clustering. This is an example of reusing an existing development graph, which uses the NFR Framework's components to capture decisions and rationale, as a resource for rapid analysis of the impact of change upon the achievement of NFRs 7. In addition, we have used domain knowledge, priorities and performance catalogues to produce customized solutions which meet needs of a particular organization.
3.3
Security Requirements
Security is an important factor, especially in medicine, and IVF is a particularly sensitive application. Security includes such concepts as integrity, confidentiality and availability 4, whose combination is used in a generic methodology for medical data security 14. For the IVF clinic, we identified two primary goals (top part of Figure 3): (1) The physical integrity of gametes of the patient is extremely crucial (indicated by "!!"). (2) The confidentiality of patient data should be maintained. A third goal is to maintain the professional integrity (reputation) of the doctor (researcher).
Physical Integrity of Patient Material. The crucial concern is that a patient's gametes must not be mistaken for someone else's. Thus, accurate identification of gametes strongly contributes to physical integrity. This can be accomplished either by using patient's name or an identifying number. Using only a number might contribute to confidentiality; for example, the lab technician, who deals with gametes, but not directly
Fig. 3. Dealing with Security Requirements for an IVF Clinic.
with patients, could in principle use only numbered dishes without knowing patient's name. However, this could increase the chance of confusing gametes of two patients, which must be avoided. Instead, the lab labels dishes with gametes using the patient's name, which is only made available to authorized personnel, including the technician. The analysis is shown in the lower left of Figure 3. It's interesting to note the interaction between the goals of physical integrity of gametes, and the confidentiality of patient information, and the resolution of the conflict to benefit the higher-priority goal. In addition, to help meet both goals, the lab has a system of physical security. While this is not shown in the figure, it is important to note that measures taken outside the computer system can have an impact on the NFRs being considered.
Confidentiality of Patient Information. The IVF clinic records some basic identifying information about a patient (name, age, hospital number, etc.), a record of the patient's visits during a treatment cycle, treatments made, and observations. In addition, the central hospital computer maintains accounting records, which do not have the details of IVF treatment. Patient information is used for both tracking individual patients for treatment, and for reviewing groups of patients for research purposes. This dual usage complicates confidentiality considerations. Furthermore, researchers sometimes need to obtain further information about particular patients, hence the statistical research information must contain some patient identifiers. Clearly, access to medical data should be restricted to authorized persons, in relation to their status 13. In the case of the IVF clinic, the mere fact that someone is an IVF patient is considered quite a personal matter, hence confidential 10. The issue of security of statistical data is a complex one. According to 32: "confusion still surrounds the question of whether privacy can be fundamentally
375
violated by statistics drawn from personal records". However, it was also shown that statistical information could provide detailed information about individuals 9. The more information pieces are tied together the more identifiable the individual is. Confidentiality of patient information must handle two goals: (1) information that identifies a patient and (2) information that does not (see Figure 3). In IVF domain, data can be used both for clinical treatment and for research. Thus, the goal of confidentiality of identifying information can be refined by the developer to handle these situations. As discussed earlier, the patient's name will be used within the lab, to meet the overriding goal of integrity of gametes, which (along with the goal of confidentiality of records) will be aided by physical security measures. To reduce the risk of names being divulged to third parties, the patient's name should not be used outside the lab. Instead, an identification number (hospital number, sample number or user generated number 16) is used. E v a l u a t i n g t h e Overall I m p a c t of Decisions. Using a name within the lab helps accurately identify gametes, and maintain its physical integrity. The selective use of name and number provides confidentiality of identifying information, both inside and outside the lab. Meeting this critical goal contributes to the overall confidentiality of patient information. In turn, meeting both that confidentiality goal and the goal for physical integrity of gametes contributes positively to maintaining the professional integrity of the doctor (researcher). While we did not initially identify professional integrity as a main goal, it is very interesting to see that the result of our analysis using the NFR Framework was in harmony with the observation that the integrity of researchers is paramount 2. 3.4
Other NFRs
Additional NFRs for the presented system include: (1) Robustness: the ability to gracefully handle a variety of situations; (2) User friendliness: providing the right degree of assistance to users; and (3) Informativeness: providing the right amount of information, appropriately arranged. Robustness concerns for the CBR system include: (1) reducing the effect of irrelevant attributes on CBR so that the prediction accuracy does not degrade with an increased number of irrelevant attributes and presenting only attributes relevant to the task; (2) fault tolerance during data entry and reasoning. Thus, the goal for robustness of the system is refined into goals for data integrity, robustness of reasoning and robustness of presentation (Figure 4, top left). Data integrity is important 14. As suggested in 13, verification and validation of data completeness and accuracy is an additional measure ensuring data integrity. Thus, especially in the early stages of system development, all attributes available should be used. This allows for correlating the attributes, which can lead to identifying data integrity violations: However, if all attributes are also used in later stages, this would lead to problems with reasoning and
376 User
RobustnessSystem
DataIntegrity System ~'~ Robustness~
Use
++
AIIAttrib.~
//
FriendlinessSystem
~
Robustness
\\
~/
I~ e U ~ ~ ~
InformativenessSystem
/
i~Chec
+
k
RetypeData Early
Later
Early
Later
Fig. 4. Dealing with Several NFRs for the System.
presentation. Thus, only relevant attributes should be used in later phases. As described in Section 2.1, knowledge-discovery techniques can be used to locate features relevant to the problem solving task 24. Using only relevant features improves flexibility 20, accuracy 22, and efficiency 23. The effect of this selective use of attributes contributes positively to the top goals of robustness and informativeness, both of which are accomplished, but user friendliness is not accomplished for the reasons described below. Generic relationships between NFRs and satisficing goals can be catalogued in advance as "correlation rules." These relationships can then be detected by the NFR assistant system, even if the developer has not explicitly linked the goals. Here, to syntactically verify data, the developer has the operator type it twice, which is helpful for data integrity. However, the NFR Assistant system detects that this is bad for user friendliness (the "correlation link," shown as a dashed line, is negative), which results in the user friendliness goal not being met. Correlation links (dashed lines) propagate values in the same way that evaluation links to.
Selecting Different Implementation Alternatives. Recognizing that system friendliness is important for users, the developer may consider ways of achieving this goal, such as implementation alternatives presented in Figure 5 (an extension of the lower left part in Figure 4). These include another user-oriented method - menu-driven input, and as system-provided checking - a dictionary of used terms, and using n-grams, which supports automatic recognition of misspelled words. In the example, n-grams are selected, so that syntactic checking remains accomplished, albeit by a different method, which contributes positively to user friendliness. In addition, the chosen methods for displaying all attributes early and relevant attributes later remain unchanged
377
from Figure 4, hence continue to contribute positively to user friendliness, which is now accomplished. Syntactic Check
/I / ~ Rele~tAttribX~ ~ ) ByUser /' Use~arly k ,t~ z / f , JAilAttri~tites (x~ (~ "N/ / ~ I ~"~/ Informativeness Robustness ~ ~) System Presentation / r
n-Grams Dictionary Menu RetypeData
Fig. 5. A Re-examination of Methods for Syntactic Checking.
This is another example of dealing with change - namely, a change in implementation alternatives. The net result is that the developer's expertise was used to accomplish the remaining top goal of user friendliness, while maintaining robustness and informativeness. This was done by reusing the information already captured in Figure 4, which dealt with several NFRs.
4
Conclusions
We are concerned with quality issues in decision support for complex information systems. We have presented an approach, called QualityCBR, for dealing with non-functional requirements for case-based reasoning systems. This integrates the NFR Framework's systematic process for building quality with the T.A3 CBR system, intended for decision support. In developing QualityCBR, catalogues have been organized to represent diverse knowledge concerning CBR, NFRs, IVF, and development techniques. By drawing on several axes (e.g., CBR and performance), we can focus on small groups of specific methods. This approach is similar to the organization of information system performance requirements issues 34. We feel that the use of such catalogues is helpful in dealing with NFRs in medical computing and other complex domains, public and private. To demonstrate how a developer can use QualityCBR to deal with conflicting and changing requirements, we illustrated its use in a medical domain. A variety of NFRs (e.g., performance, security, informativeness), and tradeoffs between individual requirements have been considered. We also found that the NFR Framework's development graphs and change facilities 7 made the process of dealing with change easier. In this paper we have considered changes in priorities of NFRs and in implementation techniques. This is consistent with results of using the NFR Framework to deal with changes in requirements for a commercial system.
378
7-,43's performance evaluation has been conducted on several domains: prediction and knowledge mining in medicine 24, 24, control task in robotic domains 21, character recognition 22, iterative browsing and intelligent retrieval 20. Each domain has different characteristics; this helps evaluation of different aspects of the system. We have evaluated both the competence 24 and scalability 23 of the system. It would be interesting to see if QualityCBR could be used to use other goaloriented approaches to requirements engineering, e.g., 8, 11, 12. This would draw on several facilities, such as representation of goals, priorities, and positive and negative links. We would like to conduct fuller studies of applying T.A3 to a variety of areas, both public and private, such as medicine, engineering and commerce, which require a variety of NFRs. Notably, we plan to explore the capability of using QualityCBR during building engineering applications, such as robotics 21, where real time response is critical. For example, the use of an "any time system" (which must produce a valid answer at any time) entails flexible and adaptive procedures to meet accuracy and safety requirements 19. These steps will help us to better asses the generality of the approach and proposed combined tools to evaluate its costs and benefits. Studies should use a methodology, such as 27 which allows us to have the kind of confidence in the results that one would have in using the scientific method. An important direction for future work is to apply CBR to the NFR Framework and its associated tool. For example, sets of development graphs for a variety of systems could be examined and analyzed to find patterns (templates) of sequences of method applications. This could be aided by facilities for critiquing and rationalizing specifications 11. Such templates could then be used as larger building blocks when using the NFR Framework to develop a variety of systems. Thus, CBR would provide underlying technology for a reuse assistant for the NFR Framework. We trust that building quality into CBR, and using CBR in tools for dealing with NFRs, will aid the development of complex information systems for a variety of public and private domains. 2 References 1. R. Agrawal, T. Imielinski, and A. Swami. Database mining: A performance pcrspective. IEEE Transactions on Knowledge and Data Eng. Learning and Discovery in Knowledge-Based Databases, 5(6):914-925, 1993. 2 Acknowledgments. The authors were with the Dept. of Computer Science, University
of Toronto, when this paper was initially prepared. This research was supported by the Information Technology Research Centre of Ontario, Canadian Consortium for Software Engineering Research and the IBM Centre for Advanced Studies. We thank Robert F. Casper and Andrea Jurisicova for much helpful information on IVF procedures. Over the years we have benefited from the insight and direction of Professors John Mylopoulos and Janice Glasgow. This paper benefits from earlier NFR work with Lawrence Chung and Eric Yu. Our families have been a constant source of support.
379
2. R. Behi and M. Nolan. Ethical issues in research. British Journal of Nursing, 4(12):712-716, 1995. 3. L. Chtmg. Dealing with security requirements during the development of information systems. In Proc. 5th Int. Conf. on Advanced Information Systems Engineering, pages 234-251, Paris, France, 1993. Springer-Verlag. 4. L. Chung. Representing and Using Non-Functional Requirements: A ProcessOriented Approach. PhD thesis, Dept. of Computer Science, Univ. of Toronto, 1993. 5. L. Chung and B. A. Nixon. Dealing with Non-Functional Requirements: Three Experimental Studies of a Process-Oriented Approach. In Proc. 17r Int. Conf. on Software Eng., pages 25-37, Seattle, WA, 1995. 6. L. Chung, B. A. Nixon, J. Mylopoulos, and E. Yu. Non-Functional Requirements in Software Engineering. In preparation, 1998. 7. L. Chung, B. A. Nixon, and E. Yu. Dealing with Change: An Approach using Non-Fhnctional Requirements. Requirements Engineering, 1(4):238-260, 1996. An earlier version, Using Non-Functional Requirements to Systematically Support Change, appeared in Proc. 2~ IEEE Int. Symp. on Requirements Eng., York, U.K., 1995, pp. 132-139. 8. A. Darderme, A. van Lamsweerde, and S. Fickas. Goal-directed requirements acquisition. Science of Computer Programming, 20:3-50, 1993. 9. D. E. Denning, P. J. Denning, and M. D. Schwartz. The tracker: A threat to statistical database security. ACM TODS, 4:76-96, 1979. 10. C.L. Early and L. C. Strong. Certificates of confidentiality: A valuable tool for protecting genetic data. American Journal of Human Genetics, 57(3):727-731, 1995. 11. S. Fickas and P. Nagarajan. Being suspicious: Critiquing problem specifications. In Proc. AAAI-88, pages 19-24, Saint Paul, MN, 1988. 12. S. F. Fickas. Automating the transformational development of software. IEEE Trans. on Software Eng., SE-11(11):1268-1277, 1985. 13. F.H. France and P.N. Gaunt. The need for security - A clinical view. International Journal of Bio-Medical Computing, 35(Suppl. 1):189-194, 1994. 14. S. Furnell, P. Gaunt, G. Pangalos, P. Sanders, and M. Warren. A generic methodology for health care data security. Medical Informatics, 19(3):229-245, 1994. 15. S. Greenspan, J. Mylopoulos, and A. Borgida. On Formal Requirements Modeling Languages: RML Revisited. In Proceedings, 16th International Conference on Software Engineering, pages 135-147, Sorrento, Italy, 1994. 16. F. Honig. When you can't ask their name: Linldng anonymous respondents with the Hogben number. Australian Journal of Public Health, 19(1):94-96, 1995. 17. W. F. Hyslop. Performance Prediction of Relational Database Management Systems. PhD thesis, Dept. of Computer Science, Univ. of Toronto, 1991. 18. M. Jarke. ConceptBase V3.1 User Manual. Univ. of Passau, 1992. 19. I. Jurisica. Supporting flexibility. A case-based reasoning approach. In The A A A I Fall Symposium. Flexible Computation in Intelligent Systems: Results, Issues, and Opportunities, Cambridge, MA, 1996. 20. I. Jurisica. Similarity-based retrieval for diverse Bookshelf software repository users. In IBM CASCON Conference, pages 224-235, Toronto, Canada, 1997. 21. I. Jurisica and J. Glasgow. A case-based reasoning approach to learning control. In 5th International Conference on Data and Knowledge Systems for Manufacturing and Engineering, DKSME-96, Phoenix, AZ, 1996.
380
22. I. Jurisica and J. Glasgow. Case-based classification using similarity-based retrieval. International Journal of Artificial Intelligence Tools. Special Issue of IEEE ICTAI-96 Best Papers, 6(4):511-536, 1997. 23. I. Jurisica and J. Glasgow. An efficient approach to iterative browsing and retrieval for case-based reasoning. In Angel Pasqual del Pobil, Jose Mira, and Moonis Aft, editors, Lecture Notes in Computer Science, IEA/AIE*98. Springer-Verlag, 1998. 24. I. Jurisica, J. Mylopoulos, J. Glasgow, H. Shapiro, and R. F. Casper. Case-based reasoning in IVF: Prediction and knowledge mining. Artificial Intelligence in Medicine, 12(1):1-24, 1998. 25. D. Leake, editor. Case.Based Reasoning: Experiences, lessons and future directions. AAAI Press, 1996. 26. D. Leake, A. Kiniey, and D. Wilson. Case-based similarity assessment: Estimating adaptability from experience. In Proc. of the AAAI-97, 1997. 27. A. S. Lee. A scientific methodology for MIS case studies. MIS Quarterly, pages 30--50, 1991. 28. J. Lee. Extending the Potts and Bruns Model for Recording Design Rationale. In Proc., 13th Int. Conf. on Software Eng., pages 114-125, Austin, Texas, 1991. 29. P. M. McShane. Customized comparative clinical results assessment of your IVF program. IVF America, 1993. 30. J. Mylopoulos, A. Borgida, M. Jarke, and M. Koubarakis. Telos: Representing knowledge about information systems. ACM Transactions on Information Systems, 8(4):325-362, 1990. 31. J. Mylopoulos, L. Chung, and B. Nixon. Representing and Using Non-Functional Requirements: A Process-Oriented Approach. IEEE Transactions on Software Engineering, 18:483-497, 1992. 32. H.B. Newcombe. When privacy threatens public health. Canadian Journal of Public Health. Revue Canadienne de Santg Publique., 83(3):188-192, 1995. 33. B. A. Nixon. Dealing with performance requirements during the development of information systems. In Proc. IEEE International Symposium on Requirements Engineering, pages 42--49, San Diego, CA, 1994. 34. B. A. Nixon. Representing and using performance requirements during the development of information systems. In Proc. ~th Int. Conf. on Extending Database Technology, pages 187-200, Cambridge, U.K., 1994. 35. B. A. Nixon. Performance Requirements for Information Systems. PhD thesis, Dept. of Computer Science, Univ. of Toronto, 1997. 36. B. A. Nixon, K. L. Chung, D. Lauzon, A. Borg'ida, J. Mylopoulos, and M. Stanley. Design of a compiler for a semantic data model. In J. W. Schmidt and C. Thanos, editors, Foundations of Knowledge Base Management, pages 293-343. SpringerVerlag, 1989. 37. J. Ortega. On the informativeness of the DNA promoter sequences domain theory. Journal of Artificial Intelligence Research, 2:361-367, 1995. Research Note. 38. H. A. Simon. The Sciences of the Artificial, 2 nd Edition. MIT Press, Cambridge, MA, 1981. 39. C.U. Smith. Performance Engineering of Software Systems. Addison-Wesley, Reading, MA, 1990. 40. P. R. Thagard, K. J. Holyoak, G. Nelson, and D. Gotchfeld. Analog retrieval by constraint satisfaction. Artificial Intelligence, 46:259-310, 1990.
Assembly Techniques for Method Engineering Sjaak Brinkkemper t, Motoshi Saeki z, Frank Harmsen 3 IBaan Company R & D, P.O. Box 143, 3770 AC Barneveld, the Netherlands, sbrinkkemper@ baan.nl 2Tokyo Institute of Technology, Ookayama 2-12-1, Meguro-ku, Tokyo 152, Japan, saeki@cs, titech.ac.jp SMoret Ernst & Young, P.O. Box 3101, 3502 GC Utrecht, the Netherlands, nlharms4 @mey.nl Abstract. As projects for developing information systems are getting larger and more complicated, we need to have more advanced development methods suitable for every development situation. Method engineering is the discipline to construct new methods from parts of existing methods, called method fragments. To achieve this objective, we need to clarify how to model the existing methods and how to assemble method fragments into new project-specific methods, so-called situational methods. Especially, to produce meaningful methods, we should impose some constraints or rules on method assembly processes. In this paper, we propose a framework for hierarchical method modelling (meta-modelling) from three orthogonal dimensions: perspectives, abstraction and granularity. According to each dimension, methods and/or method fragments are hierarchically modelled and classified. Furthermore, we present a method assembly mechanism and its formalization as a set of rules. These rules are presented in first order predicate logic and play an important role in the assembly process of meaningful methods from existing method fragments. The benefit of our technique is illustrated by an example of method assembly, namely the integration of the Object Model and Harel's Statechart into Objectcharts.
1
Introduction
The size and complexity of projects for developing information systems are becoming larger and more complicated. Therefore, development methods and supporting tools turn one of the most significant key factors to achieve great success of development projects. Until now, many methods such as structured analysis/design De Marco 78 and object-oriented analysis/design Rumbaugh91 have been proposed and many textbooks have been published. The information-technology industry is putting the existing methods and corresponding supporting tools into practice in real development projects. However, much time and effort is spent on applying the methods effectively in these projects. One of the reasons is that contemporary methods are too general and includes some parts, which do not fit to the characteristics of real projects and their contexts. To enhance the effect of methods, for each of real projects, we need to adapt the methods or construct the new ones so that they can fit to the project. Method Engineering, in particular Situational Method Engineering Harmsen 94, Brinkkemper 96 is the discipline to build project-specific methods, called situational methods, from parts of the existing methods, called method fragments. This technique is coined method assembly. In fact, many methods can be considered to be the result
382
of applying method assembly. For instance, OMT Rumbaugh 91 has been built from the existing fragments Object Class Diagram (extended Entity Relationship Diagram), State Transition Diagram, Message Sequence Chart and Data Flow Diagram, all originating from other method sources. This example shows that method assembly produced a powerful new method that could model complicated systems from multiple viewpoints: object view, behavioural view and functional view. Therefore, method assembly is a significant technique to construct both situational methods and powerful methods with multiple viewpoints. To assemble method fragments into a meaningful method, we need a procedure and representation to model method fragments and impose some constraints or rules on method assembly processes. If we allow assembly arbitrary method fragments, we may get a meaningless method. For example, it makes no sense to assemble Entity Relationship Diagram and Object Class Diagram in the same level of abstraction. Thus, the modelling technique for method fragments, so called meta-modelling technique should be able to include the formalization of this kind of constraints or rules to avoid producing meaningless methods. Several researchers applied very adequate meta-modelling techniques based on Entity Relationship Model Brinkkemper 91, Sorenson 88, Nuseibeh 95, Attribute Grammars Katayama 89, Song 94, Predicate Logic Brinkkemper 91, Saeki 94, Nuseibeh 95 and Quark Model Ajisaka 96 for various method engineering purposes (see section 6). Some of these works discuss the inconsistency of products when we assemble several methods into one, however, none of them referred to method assembly function itself yet. Song investigated existing methods, such as OMT and Ward/Mellor's Real Time SDM Ward 85, and classified the way various methods are put together Song 95. Guidelines or rules to assemble methods were not elaborated in this study. Furthermore, as discussed later in section 6, his classification is fully included in ours. In this paper, we propose a framework for hierarchical meta-modelling from three orthogonal dimensions: perspective, abstraction and granularity. According to each dimension, methods and method fragments are hierarchically modelled and classified. According to this classification of method fragments, we can provide the guideline for meaningful method assembly. That is to say, we can suggest that method fragments. which belong to a specific class can be meaningfully assembled. For example, we can sufficiently construct a meaningful method from method fragments with the same granularity level. In another example, it is not preferable to assemble the method fragments belonging to the same specific category such as Entity Relationship Diagram and Object Class Diagram, as the latter can be seen as an extension of the former. These kinds of guideline and constraints can be formalized as a set of rules based on our multiple hierarchical dimensions. These rules can be presented in first order predicate logic and play an important role on clarifying method assembly mechanism. This paper is organised as follows. In the next section, we begin with illustrating a simple example of the method fragment Statechart and introduce three orthogonal dimensions for classification of method fragments. Section 3 presents method assembly by using example of assembling Object Model and Statechart into the new
383
method fragment Objectchart. This example suggests to us what kind of guidelines or constraints are required to method assembly. We discuss these guidelines and constraints, and their formalization in section 4. Sections 5 and 6 summarize related work and our work respectively.
2
A Classification Framework for Method Fragments
2.1
Method Fragments
We begin with an example of the description of the method fragment of Harel's Statechart. Statecharts can be seen an extension of finite state transition diagram to specify reactive systems Harel 90. To avoid the explosion of the number of states occurring when we specify complicated systems with usual state transition machines, it adopted two types of structuring techniques for states, i.e. hierarchically decomposition of states: one is called AND decomposition for concurrency, and the other one is OR decomposition for state-clustering. The description of the method fragment is illustrated in the meta-model in Fig. 1 in the notation of Entity Relationship Attribute Diagrams. (To avoid confusion, we use the terms concept, association and property in method fragments instead of entity, relationship and attribute.) The Statechart technique comprises four concepts: State, Transition, Event and Firing condition. If a firing condition associated with a transition holds, the transition can occur and the system can change a state (called source state) to a destination state. During transition, the system can output or send an event to the other Statecharts. Firing conditions can be specified with predicates and/or receipt of these events. So we can have four associations among the three concepts, and two associations on the state concept for expressing AND decomposition and OR decomposition. Note that the meta-model does not include representational information, e.g. a state is represented in a rounded box in a diagram, and events are denoted by arrows. We define this kind of information as another aspect of method modelling and discuss it in the next section. AND-decomposition
I
+
Event lS SOlll'ce
of J
I
State
I
+
is destination of
:1
Transition has
~
Firing Condition
OR-decomposition
Fig. 1 Statechart Method Fragment
384
2.2
Classification of Method Fragments
Method fragments are classified according to the dimensions perspective, abstraction level, and layer of granularity. First, the perspective dimension of the classification considers the product perspective and the process perspective on methods. Product fragments represent deliverables, milestone documents, models, diagrams, etc. Process fragments represent the stages, activities and tasks to be carried out. Fig. 1 is a description of the product perspective. The abstraction dimension constitutes of the conceptual level and the technical level. Method fragments on the conceptual level are descriptions of information systems development methods or part thereof. Technical method fragments are implementable specifications of the operational parts of a method, i.e. the tools. Some conceptual fragments are to be supported by tools, and must therefore be accompanied by corresponding technical fragments. One conceptual method fragment can be related to several external and technical method fragments. The conceptual method fragment is shown in Fig. 1, whereas the corresponding technical fragment is the STATEMATE tool for specifying Statecharts Hare190. One of the most important and main discriminating properties of method fragments is the granularity layer at which they reside. Such a layer can be compared with a decomposition level in a method. A method, from the process perspective, usually consists of stages, which are further partitioned into activities and individual steps. A similar decomposition can be made of product fragments, with the entire system at the top of the tree, which is subsequently decomposed into milestone deliverables, model, model components, and concepts. Research into several applications of method engineering Brinkkemper 96 shows that methods can be projected on this classification. A method fragment can reside on one of five possible granularity layers: 9 Method, which addresses the complete method for developing the information system. For instance, the Information Engineering method resides on this granularity layer. 9 Stage, which addresses a segment of the life-cycle of the information system. An example of a method fragment residing on the Stage layer is a Technical Design Report. Another example of a Stage method fragment is a CASE tool supporting Information Engineering' s Business Area Analysis Martin 90 stage. 9 Model, which addresses a perspective Olle 91 of the information system. Such a perspective is an aspect system of an abstraction level. Examples of method fragments residing on this layer are the Data Model, and the User Interface
Model.
9 Diagram, addressing the representation of a view of a Model layer method fragment. For instance, the Object Diagram and the Glass Hierarchy both address the data perspective, but in another representation. The $tatechart resides on this granularity layer, as well as the modelling procedure to produce it.
385
Concept, which addresses the concepts and associations of the method fragments on the Diagram layer, as well as the manipulations defined on them. Concepts are subsystems of Diagram layer method fragments. Examples are:
Entity, Entity is involved in Relationship, and Identify entities
3 3.1
Method Assembly Technique Method Assembly in the Product Perspective
In this section, we introduce a simple example of method assembly - - assembling Object Model in Object-Oriented Analysis/Design and Statechart to Objectchart. Objectchart, proposed in Coleman 91, is an extension of Statechart to model reactive systems from an object-oriented view. Our framework of method assembly can explain how Objectchart was composed from the existing method fragments Object Model and Statechart. The Object Model specifies a system as a set of objects communicating with each other. Objects have their specific attributes and change their values through interobject communication. By sending messages to the other objects (or itself) an object requires of them (or itself) to provide the service that they (or it) encapsulatedly have. The objects that are requested perform their service and may change their attribute values and/or return the computed results. Objects having the same attributes and services are modelled with a Class, which is a kind of template. Fig. 2 shows the method fragment description of the Object Model at Diagram layer from conceptual level and product perspective.
f Class
has
Attribute
has
Object
Service
participatesin Association
Fig.2 Object Model Method Fragment
386
Suppose now we have to produce Objectchart by assembling these two method fragments i.e. the method models of Figs. i and 2. Fig. 3 shows the resulting method fragment of Objectchart in the same level, perspective and layer. As for this assembly process, we should note that the two method fragments belong to the same category in our three dimensional classification: conceptual level in abstraction, Diagram layer in granularity, and product in perspective. In addition we have product perspective of Objectchart in conceptual level and in Diagram Layer. Thus the method fragments with the same category can be assembled and we can get a new method with the same category.
............. S ......................................"J .............
to
.......................
ani~:~~::
~ .... ,-', .......
~'S onrlot~ted ~.th
/refers to
"~
refersto
has t FiringCondition "l Fig. 3
Objectchart : Method Assembly in the Product Perspective
The Statechart and Object Model are amalgamated to Objectchart by the following constructions: 1) A Class has a Statechart, which specifies its behaviour. 2) Attributes of a Class may be annotated to States in its Statechart. This indicates which attribute values are meaningful or visible in a specific state. 3) An Event issued during a Transition is a request of a Service to the other Object.
387
4) A Transition may change an Attribute value of an Object. The first three constructions allow us to introduce new associations "has" between Class and State, "is annotated with" between Attribute and State, and "consists of" . The concept Object participating in "consist of" stands for the object of which a service is required, i.e. a receiver of the event. Furthermore, we employ the new concept "Post condition" for specifying the change of attribute value when a transition occurs. Therefore, post conditions can define the effect of service-execution on attributes. Let's explore what manipulations were made and what kinds of constraints could be considered in this example. The basic manipulations that we applied here are: 1) Addition of a new concept (Post condition), 2) Addition of a new association (is_annotated_with, consists_of, has), 3) Addition of a new property (is_hidden). First of all, when we assemble two method fragments, we should introduce at least one new concept or association. If we did not introduce anything, it would mean that a method fragment was completely included in another one. This case might be meaningless because we could not find the effect of this method assembly and the result was the same as the containing method fragment. This applies for the meaningless example of assembling ERD and Object Class Diagram (the super class of ERD), which we mentioned in section 1. Furthermore, at least one connection between the two method fragments through newly introduced associations and/or concepts should be introduced, because the two method fragments are to be conceptually connected by the method assembly. Consequently, these constraints can be generalized as Rule 1)
At least one concept, association or property should be newly introduced to each method fragment to be assembled, i.e. a method fragment to be assembled should not be a subset of another.
Rule 2)
We should have at least one concept and/or association that connects between two method fragments to be assembled.
Rule 3)
If we add new concepts, they should be connectors to both of the assembled method fragments.
Rule 4)
If we add new associations, the two method fragments to be assembled should participate in them.
The following additional rules can easily be determined, whose explanation we omit. Rule 5)
There are no isolated parts in the resulting method fragments.
Rule 6)
There are no concepts which have the same name and which have the different occurrences in a method description.
These rules apply for method fragments in the conceptual level and diagram layer. If the method fragment to be assembled is related to the other levels or layers, the effect
388
of assembly propagates to the others. It means that we should have the other types of rules. For example, the different concepts on the conceptual level should have different representation forms (notation) on the technical level. We will discuss a more elaborated style of rules and their formalization in section 4.
3.2
Method Assembly in the Process Perspective
In the previous example, we illustrated product-perspective method assembly. Next, we turn to discuss the process-perspective method assembly also with the help of an example. Suppose we have the process descriptions for Object Model and for Statechart in Diagram layer at our disposal, e.g. for Object Model:
Draw an Object Model O1) Identify objects and classes, 02) Identify relationships, 03) Identify attributes and services. and for Statechart:
Draw a Statechart $1) Identify states, $2) Identify state changes and their triggers, $3) Cluster states, and so on. According to Coleman 92, the recommended procedure for modelling Objectcharts is as follows:
Draw an Objectchart OCI) Draw an Object Model, OC2) For each significant class, Draw a Statechart, and OC3) Refine the Statechart to an Objectchart by adding post conditions and annotating states of the Statechart with attributes. This procedure is constructed from the two process method fragments, Object Model (step OC1)) and Statechart (step OC2)) and seems to be natural. In more detail, between steps OC1) and OC2), we find that we should perform the activity of identifying the relationship "has" between Class and State shown in the Fig. 3. The concept "Post condition" and its associations, say "refers to" , and the association "is annotated with" are identified while the step OC3) is being performed. It means that newly added concepts and associations to connect the product-perspective method fragments to be assembled should not be identified until the associated concepts are identified. In fact, it is difficult for us to identify the association "has" between classes and states before we have identified classes or identified states and we should avoid this execution order of the activities (see also Fig. 4).
389
Rule 7)
The activity of identifying the added concepts and relationships that are newly introduced for method assembly should be performed after their associated concepts are identified.
The rule mentioned above provides a criterion to make meaningful and useful procedures from manipulations on concepts and associations in Diagram Layer. Similarly, we can easily have the rule : we should not identify any associations until we identify their associated concepts in Diagram Layer. So the first step of method procedure should be identifying some concepts. This results from the natural execution order of human perception.
/
~t
I
S 1: Identify States I
$2: Identify State changes and Triggers
I
02: Identify Associations I I
Diagram with/ lasses and/ ssociatio~fs
tate Transitio/n/ Diagram/
$3: Clustering States ...
Statechart /
OCt: Draw an Object Model(A)
OC2: Draw a Statechart (B) OC3: Refine Statecharts /Objectcha~
Draw an Objectchart (C)
Fig. 4 Method Assembly in the Process Perspective
Another type of rules relates to the input/output order of products to activities. For example, the activity step 02) in Object Model consumes the identified objects and classes as its inputs which are produced by the step O1). The point in method assembly processes is what input-output relationships are added and/or changed. In this example, as shown in Fig. 4, the step OC2) in Objectchart, which resulted from steps S1), $2) and $3) in Statechart, should consume the identified classes as its
390
inputs. They are the output of the step O1) in Object Model, i.e. another method fragment. Therefore we can have the following rule: Rule 8) Let A and B be the two method fragments to be assembled, and C the new method fragment. In C, we should have at least one product which is the output of A and which is the input of B, or the other way round. This rule means that either of the method fragments to be assembled, say A, should produce input to the activities of B in the new method C. More examples of method assembly rules in process perspective will be shown in section 4.
3.3
Discussion of Method Assembly on Three Dimensions
As we have shown in section 2, method fragments can be considered on three dimensions: perspective, abstraction level and granularity layer. These dimensions can be used to improve, speed up, and simplify the method assembly process. We illustrate this with the following example. Assembling Object Model and Statechart, which are product fragments at the Diagram layer and at the conceptual level, implies the assembly of method fragments addressing the other perspective, abstraction level, and granularity layers. Associated with the Statechart and Object Model product fragments are modeling procedures, i.e. process fragments. The assembled modeling procedure results from the components of each of these two process fragments. Some of the rules that apply are: Rule 9) Each product fragment should be produced by a "corresponding" process fragment. Rule 10) Suppose a product fragment has been assembled. The process fragment that produces this product fragment consists of the process fragments that produce the components of the product fragment. Also associated with the conceptual method fragments mentioned above are technical method fragments, such as Object Model and Statechart diagram editors, a repository to store object models and Statecharts, and a process manager to support the modeling procedures for object models and Statecharts. Similarly, the assembly of these technical method fragments results from the assembly of the corresponding conceptual method fragments: Rule 11)
A technical method fragment should supports a conceptual method fragment.
The assembly of fragments at the Diagram layer has also implications for the components of these fragments, which are at the Concept layer. In general, assembly of two method fragments results in the assembly of method fragments of lower granularity layers. As we have seen in section 3.1, the assembly of Object Model and Statechart results in the assembly of Service and Event, Class and State, and Attribute and Firing Condition. A rule that applies to this is: Rule 12)
If an association exists between two product fragments, there should exist at least one association between their respective components
391
We have taken in the above example the assembly of conceptual product fragments at the Diagram layer as a starting point. However, the starting point can be at any combination of perspective, abstraction level, and granularity layer. Obviously, whatever starting point is used, the result of one assembly action is a cascade of other actions within the three-dimensional framework.
4 4.1
Method Assembly : Guideline and Formalization Requirements for Method Assembly
Method assembly should ensure that the selected method fragments are mutually adjusted, i.e. they have to be combined in such a way that the resulting situational method does not contain any defects or inconsistencies. Several types of defects can appear: 9 Internal incompleteness, which is the case if a method fragment requires another method fragment that is not present in the situational method. For instance, a data model has been selected without the corresponding modelling procedure and tool. 9 Inconsistency, which is the case if the selection of a method fragment contradicts the selection of another method fragment. For instance, two similar data modelling techniques have been selected without any additional reason. 9 Inapplicability, which is the case if method fragments cannot be applied by project members, due to insufficient capability. All these issues relate to the internal or situation-independent quality Hoef 95 of a situational method, i.e. the quality of a method without taking into consideration the situation in which the method is applied. The two most important criteria are: 9 Completeness: the situational method contains all the method fragments that are referred to by other fragments in the situational method. 9
Consistency: all activities, products, tools and people plus their -mutualrelationships in a situational method do not contain any contradiction and are thus mutually consistent.
Furthermore, we distinguish the following method internal quality criteria that are not treated in this paper for the sake of brevity and their details is in Harmsen 97: 9 Efficiency: the method can be performed at minimal cost and effort 9 Reliability: the method is semantically correct and meaningful 9 Applicability: the developers are able to apply the situational method The effort to achieve situation-independent quality of method fragments is considerable. Method fragments can be combined in a lot of ways, many of which are meaningless. Moreover, method fragments require other method fragments to be meaningful in a situational method, or require certain skills from the actors related to them. This is illustrated by the following small example. Suppose a process
392
perspective method fragment Draw an Object Model (shown in sect. 3.2) has been selected. The following should be at least verified ; 1) No similar method fragment already exists in the situational method, 2) The specification of the Object Model produced by the process fragment is selected, 3) Actors have the expertise to deal with this process fragment, and 4) The products required are produced by preceding selected process fragments (See also the examples in sect. 3.1 and sect. 3.2). Internal method quality can only be achieved by a set of guidelines on the Method Engineering level. These formalized guidelines are presented in the form of axioms, which can be considered an extension of the set of axioms, corollaries and theorems presented in section 4. The axioms are grouped by the various quality criteria.
4.2
Classification of Method Assembly
In this section, the general internal quality requirements completeness and consistency are further partitioned by means of the three-dimensional classification framework. Completeness is partitioned into: 9
Input/output completeness, stating that if a process fragment requiring or manipulating a product fragment is selected, then that product fragment should be available in the situational method. Input/output completeness applies to the interaction of the two perspectives.
9
Content completeness, stating that if a method fragment is selected, all of its contents have to be available too. Contents completeness applies to the relationship between granularity layers.
9
Process completeness, requiring that all product fragments have to be, in some way, produced. Process completeness is related to the interaction of the two perspectives.
9
Association completeness, requiring that product fragments on certain layers are always involved in an association, and that associations always involve product fragments. Association completeness relates to the product perspective.
9
Support completeness, requiring that technical method fragments support conceptual method fragments. Support completeness applies to the relationship between abstraction levels. Consistency is partitioned into:
9
Precedence consistency, requiring that product fragments and process fragments are placed in the right order in the situational method. This type of consistency applies to the interaction between perspectives.
9
Perspective consistency, requiring that the contents of product fragments is consistent with the contents of process fragments. Perspective consistency also applies to the interaction between perspectives.
393
9
Support consistency, requiring that technical method fragments are mutually consistent. Support consistency relates to the relationships of technical method fragments.
9
Granularity consistency, which imposes that the granularity layers of related method fragments are similar, and that their contents are mutually consistent. This type of consistency applies to the interaction between granularity layers.
9
Concurrence consistency, which requires parallel activities to be properly synchronized. Concurrence consistency relates to the interaction of process fragments.
Note that our concepts of "completeness" and "consistency" are syntactical constraints on descriptions of method fragments written in Entity Relationship Model. To formalize actual method assembly processes more rigorously and precisely, we should consider some aspects of the meaning of method fragments. In the example of Objectchart, we associated the concept "Attribute" with "State". The question is in whatever method assembly we can always do it. The answer depends on the semantics of these concepts in the method fragments. How to specify the semantics of method fragments for method assembly is one of the most important and interesting future topics. In the next sub-section, each of these categories will be elaborated by means of an example taken from the Objectchart case.
4.3
Method AssemblyRules
4.3.1 Some Definitions As noticed before, the natural language representation of method assembly rules creates some problems regarding ambiguity and implementability. Therefore we have formalized our theory regarding method fragments, and expressed the rules in that formalization. In this sub-section, we only show the part of the formalization required in the context of this paper. Moreover, we give examples of rules, some of which are formalized well. The formalization employs the following notions:
Set, which represents a category of similar method fragments. Predicate, which represents a relationship between Method Base concepts. Function, which represents the assignment of the method fragment properties to method fragments The usual logical quantifiers and operators. The operators <, =, ~, c , u and c~. The following sets are defined:
394
M = C u T, the set of method fragments C = R u P, the set of conceptual method fragments: e.g. Draw an Object Model, Object Model, Statechart, Identify Clas~es and Objects, Class, Object, Service, Transition "has" Event, List of States. R the set of product fragments, e.g. Object Model, Statechart, List of States P the set of process fragments, e.g. Class, Object, Service, State, Event. CN c_ R , the set of concepts, e.g. Class, Object, Service, State, Event.of concepts are
postulated A c R, the set of associations, e.g. Transition "has" Event, State "is annotated with" Attribute. T the set of technical method fragments. If a method fragment is selected for inclusion in a situational method, it is indexed with an "s" , for instance: Rs is the set of selected product fragments. The following predicates are used in this section: 9
contents and contents* c_ R X R u P X P to represent the non-transitive and transitive consists-of relationship between method fragments, e.g. contents(Class, Object Model);
9
manipulation c_ P X R ,
9
involvement c A XR, to represent the fact that an association involves a product fragment, e.g. involvement (is annotated with, Objectchart);
to represent the fact that a process fragment manipulates (i.e. produces, updates, etc.) a certain product fragment, e.g. manipulation(Draw an Objectchart, Objectchart);
9 prerequisite c_ P X R, to represent the fact that a process fragment requires a product fragment for its execution, e.g. prerequisite(Identify Associations, List of
Classes and Objects); 9 precedence c_ P X P, denote the precedence relationship between process fragments, e.g. precedence(Identify Associations, Identify Classes and Objects); 9
support ~ C • T, to represent that a technical method fragment supports a conceptual method fragment, e.g. support(Statechart, STATEMATE);
9
concurrence, to represent the fact that two process fragments can be performed in parallel, e.g. concurrence(Identify Associations(O2), Identify States(S1)) (see Fig.4).
9
layer." M ~ Method, Stage, Model, Diagram, Concept,to return the layer of the fragment (see sect. 2.2), e.g. layer(Objectchart)=Diagram, laye r( Class )=Concept.
method
395
Below, each type of completeness and consistency, as defined in sect. 4.1, is related to our Objectchart example. We assume that both Object Model, Statechart, and Objectchart should be part of a complete and consistent situational method, M s
Step 2 of the Objectchart modeling procedure requires an Object Model. The description of the Object Model should therefore exist in the situational method. In general, the rule is: Required product fragments should have been selected for the method assembly, i.e. Vp ~ Ps, r ~ R prerequisite(p, r) --> r e R s
Contents completeness
Concepts (product fragments) such as Class, Object, State, Service, Transition etc. should always be part of another product fragment. Note that this is indeed the case, as they are all components of Statechart. In a formalized way, this rule is defined as follows: Vr1 ~ Rs3r 2 ~ Rslayer(r 1 ) = concept --->contents * (r2 , r1) A layer(r2 ) ~ {Model, Diagram}
Process completeness
Suppose the Objectchart is included in the situational method. Then it has to be produced by some process fragment that is also included. In general, selected product fragments at the lowest four granularity layers have to be produced by a selected process fragment, i.e. Vr ~ Rs3 p ~ Ps layer(r) ~ Concept --->manipulation(p, r)
Association completeness
Suppose both the Object Model and State Chart have been selected for inclusion in the situational method. Then they should be connected by at least one association (note, again, that this is the case; they are connected by even more than one association). In general, if more than one diagram layer product fragment has been selected, diagram layer product fragments should be associated with at least one other diagram layer product fragment. (Rule 4)).
396
Vr1 , r2 ~ R s 3 a ~ A s layer(r 1 ) = Diagram A layer(r 2) = Diagram A r1 ~ r 2 --->involvement(a, r1 ) A involvement(a, r2) Also Rule 3) is an example of an association completeness rule: V r l , r 2 ~ RsBal ,a 2 ~ As3c ~ CN s(layer(rl ) = Diagram^ layer(r2) = Diagram ^ r1 ~ r2 ) --> involvement(a 1 , r1 ) A involvement(a 2 , r2 ) A involvement(c, r1 ) A involvement(c, r2 ) From these rules we can deduce, that Rule 2) is redundant. Support completeness Suppose the STATEMATE editor was selected for inclusion in our situational method. Then, the Statechart product fragment that is supported by this editor should also be included. In a formalized way, this rule, i.e.Rule 11) is defined as follows: Vt ~ Ts,r ~ Rsupport(r,t)
--+ r e R
4.3.3 Consistency Rule Precedence consistency In the modeling procedure for Objectchart, step OC2 requires an Object Model. This Object Model should be produced by a step before step OC2. In general: a process fragment producing a required product fragment should be placed before the process fragment requiring the product fragment, i.e. '~Pl e Ps ,r e Rs3P2 e P prerequisite(P1, r) ---> manipulation(p 2 , r) ^ precedence(P1, P2 ) This rule is a part of Rule 7). This rule means that we should have at least one new process fragment and this new fragment should not be first in the order of the assembled process fragments. In the example of Fig. 4, we have a new process fragment "Refine Statechart (OC3)" , and it cannot be performed before Draw an Objectchart and Draw a Statechart. The above rule specifies the latter part. We can also formalize the former part. Perspective consistency Objectchart is produced by the modeling procedure presented in section 3.2. The components of Objectchart, its concepts, should be produced by components of this fragment. As a general rule: If a product fragment is produced by a certain process fragment, then all of its contents should be produced by the sub-processes of that process fragment, i.e.
397
V P l ' P2 E P s , r ~ R s , b ~ B 3 r 2 ~ R s m a n i p u l a t i o n ( p ---> contents(r 1 , r2 ) A m a n i p u l a t i o n ( P 2 , r2 )
I , r I ) A c o n t e n t s ( P l , p2 )
Granularity consistency
An example of a granularity consistency rule is Rule 12) (section 3.4), stating that if two product fragments are associated, there should be at least an association at the Concept layer in their perspective contents as well, i.e.: Va I ~ As,rl,r 2 ~ Rs,ll,l 2 ~ L3Cl,C 2 ~ CNs,a 2 ~ A s involvement( a 1 , r1 ) ^ involvement( a 1 , r 2 ) contents * (r1 , c 1 ) ^ contents * (r 2 , c 2 ) A involvement(a 2 , c 1 ) ^ involvemnet(a 2 , c 2 )
Concurrence consistency
Suppose the Objectchart process fragment consists, to speed up the process, of two steps that are concurrently executed. This may only be the case, if they do not require complete products from each other. So, for instance, steps OC1 and OC2 of the Draw an Objectchart fragment may not be concurrently executed, as step OC2 required some intermediate results produced by step OC1. However, within this fragment some steps can be performed concurrently, e.g. 02 and S1. The concurrence consistency rule is defined as follows: V P l ' P 2 ~ Ps, r ~ R s c o n c u r r e n c e ( P l , P 2 ) ~ ( prerequisite( P l , r) A m a n i p u l a t i o n ( P 2 , r ) ) A ~ ( prerequisite( P 2 , r) A manipulation( P l , r))
5
Related Work
As mentioned before, several meta-modelling techniques were proposed, e.g. they were based on Entity Relationship Model, Attribute Grammar, Predicate Logic and Quark Model. Comparison of meta-modelling techniques and their languages was also discussed in Harmsen 96. We pick up a few representatives and discuss their relevance to our work. Almost all approaches to meta-modelling are using Entity Relationship Model (ER). Some applied Predicate Logic to describing the properties, which cannot be represented with just the ER notation. For instance, the Viewpoints approach Nuseibeh 92 combines ER and Predicate Logic. It aims at constructing a method with multiple views from the existing methods. In other words, we can define the assembly mechanism of the products, which are produced by the different existing methods. The approach also provides the function for defining constraints to maintain consistency on the products that are produced by the existing methods. However, it discusses about the constraints on the assembled products but not constraints on method assembly processes themselves.
398
Software Quark Model Ajisaka 96 tried to formalize a restricted set of atomic concepts, which can specify any kind of software products and it can be considered as a product perspective of meta-modelling. The aim of the model seems to be not method assembly in product level, but maintaining causality relationships among the software products produced in various stages of a software development cycle through atomic concepts. In his article, Song investigated the existing integrated methods, into which several different methods were integrated, and classified method integration from benefitoriented view, i.e. classification criteria is based on what benefit we can get by the integration Song 95. He did not use the term "'assembly" but "'integration". According to his classification, we can have two categories: function-driven (a new function is added) and quality-driven (the quality of a method is improved). He also classified these two categories in detail based on which components of methods are integrated, e.g. Artifact Model Integration, Process Integration, Representation Integration and so on. His work is a pioneer of method assembly research. However, he did not discuss how to integrate (assemble) methods or what rules should hold for each category but just classified the existing integration patterns. And, all of his proposed classes are not necessary orthogonal, i.e. an integration is included in several classes. Our framework is completely orthogonal and we have shown some guidelines and rules to produce meaningful methods. Furthermore our classification includes Song's classification. Fig. 3 is an example of Song's Artifact Model Integration, i.e. method assembly in Conceptual Level, Product Perspective and Diagram Layer.
6
Conclusion and Future Work
This paper clarifies how to assemble method fragments into a situational method and formalize rules to construct meaningful methods. We have already extracted over 80 rules thought real method assembly processes. Our rules are general ones which are applicable for arbitrary method assembly, and we may need some rules for specific kinds of method assembly. These rules probably include semantic information on method fragments and on systems to be developed. Our next goal is to assess our generic rules in more complicated and larger-scale assembly processes, e.g. whether our rules are sufficient and minimal to specify method assembly processes as general rules, and to look for specific rules as method assembly knowledge. Our rules are described with predicate logic, so we have a possibility to check method fragments automatically during the assembly processes. To get efficient support, we should consider how our rules can be efficiently executed in our method base system, which stores various kinds of method fragments. As reported elsewhere, we are currently developing the Computer Aided Method Engineering (CAME) tool, called Decamerone Harmsen 95, which includes a comprehensive method base system. A support function for method assembly processes based on our assembly rules is currently under development. Functionality for adaptive repository generation and customisable process managers is being realised. Next to this, the Method Engineering Language (MEL) is under development Harmsen 96. This language allows us to describe method fragments from the various relevant dimensions.
399
Operators for the manipulation, storage and retrieval of method fragments in the method base have been defined. To clarify which method fragments are suitable and useful for a specific situation is one of the most important research issues and empirical studies are necessary such as Slooten 96 and Klooster 97.
References Ajisaka 96
Ajisaka,T.. The Software Quark Model: A Universal Model for CASE Repositories. In Journal of Information and Software Technology, 1996.
Brinkkemper 94
Brinkkemper, S., Method Engineering: Engineering of Information Systems Development Methods and Tools. In Journal of Information and Software Technology, 1996.
Coleman 92
Coleman,F., Hayes,F. and Bear,S., Introducing Objectcharts or How to Use Statecharts on Object-Oriented Design. IEEE Trans Soft. Eng., Vol.18, No.l, pp.9 -- 18, 1992.
De Marco 78
DeMarco, T., Structured Analysis and System Specification, Yourdon Press, 1978.
Harel 90
Harel,D., Lachover,H., Naamad.A., Pnueli,A., Politi,M., Sherman,R. Shutull-Trauring,A. and Trakhtenbrot,M., STATEMATE: A Working Environment for the Development of Complex Reactive Systems. IEEE Trans. Soft. Eng., Vol.16, pp.403 -- 414, 1990.
Harmsen 94
Harmsen, F., S. Brinkkemper, H. Oei, Situational Method Engineering for Information System Projects. In: Olle, T.W., and A.A. Verrijn Stuart (Eds.), Methods and Associated Tools for the Information Systems Life Cycle, Proceedings of the IFIP WG8.1 Working Conference CRIS' 94, North-Holland, pp. 169-194, Amsterdam, 1994.
Harmsen 95
Harmsen, F. and S. Brinkkemper, Design and Implementation of a Method Base Management System for a Situational CASE Environment. In: Proceedings of the APSEC' 95 Conference, IEEE Computer Society Press, Los Alamitos, CA, 1995.
Harmsen 96
Harmsen, F., and M. Saeki, Comparison of Four Method Engineering Languages. In: In: S. Brinkkemper, K. Lyytinen and R. Welke (Eds.), Method Engineering: Principles of Method Construction and Tool Support, Chapman & Hall, pp.209-231, 1996.
Harmsen 97
Harmsen, F., Situational Method Engineering. Moret Ernst & Young, 1997
400 Hoef 95
Hoef, R. van de, and F. Harmsen, Quality Requirements for Situational Methods. In: Grosz, G. (Ed.), In Proceedings of the Sixth Workshop on the Next Generation of CASE Tools, Jyv/~skyl/~,Finland, June 1995.
Katayama 89
Katayama, T., A Hierarchical and Functional Software Process Description and Its Enaction. In: Proceedings of 11t~ Int. Conf. on Software En~neering. pp.-343-352, May 1989.
Klooster 97
Klooster, M., S. Brinkkemper, F. Harmsen, and G. Wijers, Intranet Facilitated Knowledge Management: A Theory and Tool for Defining Situational Methods. In: A. Olive, J.A. Pastor (Eds.), Proceedings of CAiSE'97. Lecture Notes in Computer Science 1250, Springer Verlag, pp.303-317, 1997.
Nuseibeh 95
Nuseibeh, B., J Kramer and A. Finkelstein, Expressing the Relationship between Multiple View in Requirements Specification. In: Proceedings of 15th Int. Conf. on Software Engineering, Baltimore, IEEE Computer Society Press, pp. 187197, 1993.
Olle 91 j
OUe, T.W., J. Hagelstein, I.G. MacDonald, C. Rolland, H.G. Sol, F.J.M. van Asssche, A.A. Verrijn-Stuart, Information Systems Methodologies - A Framework for Understanding, 2na Edition, Addison-Wesley, 1991.
Saeki, M., and K. Wen-yin, Specifying Software Specification and Design Methods. In: G. Wijers, S. Brinkkemper, T. Wasserman (Eds.), Proceedings of CAiSE'94, Lecture Notes in Computer Science 811, Springer Verlag, pp. 353-366, Berlin, 1994.
Slooten 96
Slooten, K. van and B. Hodes, Characterizing IS Development Projects. In: S. Brinkkemper, K. Lyytinen and R. Welke (Eds.), Method Engineering: Principles of Method Construction and Tool Support, Chapman & Hall, pp.29-44, 1996
Song 95
Song, X., A Framework for Understanding the Integration of Design Methodologies. In: ACM SIGSOPT Software Engineering Notes, Vol. 20, No. 1, pp. 46-54, 1995.
Sorenseon 88
Sorenson,P.G., J.P.Tremblay, A.J.McAllister, The Metaview System for Many Specifications Environements. In IEEE Software, Vol.30, No.3, pp.30-38, 1988.
Ward 85
Ward,P, S. Mellor, Structured Development for Real-time Systems, Yourdon Press, 1985.
Formalizing Materialization Using Metaclass Approach * Mohamed
a
Dahchour t
Abstract Materialization is a powerful and ubiquitous abstraction pattern for conceptual modeling. Intuitively, it relates a class of categories (e.g., models of cars) and a class of more concrete objects (e.g., individual cars). This paper formalizes the semantics of materialization using the metaclass approach of the TELOS data model. Formulas can be uniformly attached to classes, metaclasses, and meta-attributes to enforce integrity constraints and deductive rules relevant to materialization semantics. The paper also proposes some suggestions for extending TELOS to capture some materialization semantics which cannot be represented with the available constructs. Keywords: Object Orientation, Materialization Relationship, Metaclass, TELOS.
1
Introduction
Conceptual modeling is the activity of formalizing some aspects of the physical and social world around us for purposes of understanding and communication. Generic relationships are powerful abstraction constructs that help narrow the gap between concepts in the real world and their representation in conceptual models. For full benefit, these relationships should be made available in objectoriented languages and systems as primitives for developing conceptual models of applications. However, before their implementation, we believe that generic relationships should be first well formalized. This formalization will eliminate *This work is part of the YEROOS (Yet another Evaluation and Research on Object-Oriented Strategies) project, principally based at the University of Louvain. See http://yeroos.qant.ucl.ac.be. ?University of Louvain, INGI (Department of Computer Science and Engineering), 2 Place Sainte-Barbe, 1348 Louvain-la-Neuve, Belgium, e-maih dahchour~student.fsa.ucl.ac.be
402 the possible ambiguities between similar relationships and will play an intermediate role between the informal description of a relationship and its factual implementation. This paper presents a formalization of materialization PZMY94. Materialization is a powerful and ubiquitous abstraction pattern. It is a semantic relationship between a class of abstract categories (e.g., models of cars) and a class of more concrete objects (e.g., individual cars). The semantics of materialization concern both classes and instances of these classes. Consequently, the formal specification of materialization must include both the specification of the class and the instance levels in a coordinated manner KS95. Furthermore, constraints associated with generic relationships must be defined at the conceptual level, since they govern all instances of these relationships. We remove, therefore, the burden from the designers who otherwise would have to define these constraints for each realization of materialization. We use the metaclass approach of TELOS, a language for representing knowledge about information systems MBJK90, to formalize materialization. T E L O S has already been used to partially formalize semantics of partOf IMP93 and memberOf MPK96 relationships. The metaclass approach has been used successfully to implement some generic relationships (see e.g., HGPK94, KS95, GSR96). Particularly, in our previous work DPZ97, we have presented three metaclass approaches to implement generic relationships and in DPZ96, we have used one of these approaches to implement materialization in an abstract target system. In this paper, we use the metaclass approach of T E L O S for the formalization purpose. The paper is organized as follows. Section 2 gives an overview of materialization. Section 3 presents the main features of the T E L O S data model, relevant to our formalization. Sections 4 and 5 formalize in detail the semantics of materialization at both the class and instance levels. Section 6 summarizes and concludes the paper.
2
Materialization
This section gives an overview of the materialization relationship and of its specific attribute propagation mechanisms. More detail can be found in PZMY94. 2.1
Intuitive
definition
Intuitively, materialization relates a class of categories to a class of more concrete objects. Figure l(a) shows a materialization relating two classes: class CarModel has two monovalued attributes (name and sticker_price) and four multivalued attributes (#doors, eng..size, auto_sound, and special_equip); class Car defines three monovalued attributes (manuf_date, serialS, and owner). CarModel represents information typically displayed in the catalog of car dealers (namely,
name= Flat-rctm suckar_price=l 0.000 I #doors={3,5} I eng..size= 1200,1300} auto_sound={ tape, radio} specml_eqolp= {atrbag, alarm. k cru,se} ,~
Nico's Fiat
name = Fint-rctro s.cker..pnce= 10,000 #doors= 3
I eng_s~e~ 1200 auto_sound~ {lape, radio} aitbag=Acm.e alarm=Burglar_lOng chase= Fiat manuf_date= 111195 serial#= 123 owncl~- Nlr j
Figure 1: An example of materialization. name and price of a car model, and lists of options for number of doors, engine size, sound equipment, and special equipment). Car represents information about individual cars (namely, manufacturing date, serial number, and owner identification). As in PZMY94, we draw a materialization link as a straight line with a star 9 on the side of the more concrete class. Figure l(b) shows an instance FiatRetro of CarModel and an instance Nico's Fiat of Car, of model FiatRetro. CarModel is the more abstract 1 class and Car is the more concrete class of materialization CarModel--*Car. Intuitively, this means that every concrete car (e.g., Nico's Fiat) has exactly one model (e.g., FiatRetro), while there can be any number of cars of a given model. Further intuition about abstractness/concreteness is that each car is a concrete realization (or materialization) of a given car model, of which it "inherits" a number of properties in several ways. Nico's Fiat thus directly inherits the name and sticker_price of its model FiatRetro; this mechanism is called Type 1 attribute propagation. Nico's Fiat has attributes #doors, eng_size, and auto_sound whose values are selections among the options offered by multivalued attributes with the same name in FiatRetro; this is called Type 2 attribute propagation. For example, the value {1200,1300} of eng_size for FiatRetro indicates that each FiatRetro car comes with either eng_size = 1200 or eng_size = 1300 (e.g., 1200
for Nico's Fiat). The value {airbag, alarm, cruise_ctrl} of attribute special_equip for FiatRetro means that each car of model FiatRetro comes with three pieces of special equipment: an airbag, an alarm system, and a cruise control system. Thus, Nico's Fiat has three new attributes named airbag, alarm, and cruise_ctrl, whose suppliers are, respectively, Acrne, Burglar_King, and Fiat. Other FiatRetro cars might have different suppliers for their special equipment. This mechanism is called Type 3 attribute propagation. In addition to those attributes propagated from the instance FiatRetro of class CarModel, Nico's Fiat of course has a 1The notion of abstractness/concreteness of materialization is distinct from the notion of abstract class of object models, where an abstract class is a class without instances, whose complete definition is typically deferred to subclasses.
404 value for attributes manuf_date, serial#, and owner of class Car. The semantics of attribute propagation is defined more precisely in Section 2.3. Abstract classes can materialize into several concrete classes. For example, data for a movie rental store could involve a class Movie, with attributes director, prod.cer, and year, that materializes independently into classes VideoTape and VideoDisc (i.e., VideoTape*--Movie--,VideoDisc). VideoTapes and VideoDiscs could have attributes like inventory#, system (e.g., PAL or NTSC for VideoTape), language, availability (i.e., in-store or rented), and so on. Materializations can also be composed in hierarchies, where the concrete class of one materialization is also the abstract class of another materialization, and so on (e.g., Play--*Setting--*Pefformance). For the sake of space, this paper considers only simple materialization hierarchies A--*C and abstract classes materializing in more than one concrete class as in CI*---A--*C2. A complete formalization of materialization, including composition of materializations, can be found in Dah97. 2.2
Semi-formal
semantics
We now summarize the necessary elements for a semi-formal definition of materialization. Materialization is a binary relationship (A- *C) between two classes A and C, where A is more abstract than C (or C is more concrete than A). Most real-world examples of materializations have cardinality 1,1 on the side of the more concrete class C and cardinality 0, N on the side of the more abstract class A. Application semantics can further constrain the cardinality of the A-side to Cmin, Cmax, with the meaning that at least Crn~n and at most Cmax concrete objects are associated with each abstract object.
~bject
I a c ~
|
{al
|
~ T~-f~t ef~'et d~
~~~"~.cla$s|acet Two-f~d object facet ....
clM. facet
(b)
Figure 2: Semantics of materialization. The semantics of materialization is conveniently defined as a combination of usual is-a (generalization) and is-o/(classification), and of a class/metaclass correspondence. Figure 2(a) shows how the semantics of materialization A--*fi is expressed with a collection of two-/aceted constructs. Each two-faceted construct is a composite structure comprising an object, called the object facet, and an associated class, called the class acet. The object facet is an instance of the more abstract class A, while the class facet is a subclass of the more concrete
405 class C. The semantics of materialization induce a partition of C into a family of subclasses {Ci}, such that each Ci is associated with exactly one instance of A. Subclasses Ci inherit attributes from C through the classical inheritance mechanism of the is-a link. They also "inherit" attributes from A, through the mechanisms of attribute propagation described in the next section. Objects of C, with attribute values "inherited" from an instance of A, are ordinary instances of the class facet associated with that instance of A. As in Figure 1, we draw classes as rectangular boxes and instances as rectangular boxes with rounded corners. Classification links (is-of) appear as dashed arrows, and generalization links (is-a) as solid arrows. To underline their double role, we draw a two-faceted construct as an object box adjacent to a class box. Figure 2(b) sketches the basic semantics of the materialization of Figure l(a). The FiatRetro instance of CarModel is the object facet of a two-faceted construct, whose class facet is the subclass FiatRetro_Cars of Car, describing all instances of Car with model FiatRetro. For users, Nico's Fiat and John's Fiat are instances of Car. Our semantics and its formalization describe them as ordinary instances of FiatRetro_Cars. Wild_2CV is another instance of CarModel and Guy's 2CV is an instance of class facet Wild_2CV_Cars.
2.3
Attribute propagation
Attribute propagation from the more abstract class to the more concrete class of a materialization is precisely defined as a transfer of information from an abstract object to its associated class facet in a two-faceted construct, as illustrated in Figure 3. The three mechanisms of attribute propagation are defined precisely as follows: CarModel name (T1)
Nico's Fiat name=Fiabreu'o stiekar_price=10.000 #doors=3 eng_size= 1200 auto-sound={tape,radio } airbag = Acme alarm Burglar King eraise = Fiat manuf_date= 1/1/95 sarial#=123 owner=NICO
=
Figure 3: Attribute propagation between CarModel and Car.
406 1. For users, Type 1 propagation characterizes the plain transfer of an attribute value from an instance of the abstract class to instances of the concrete class. In our semantics, the value of a (monovalued or multivalued) attribute is propagated from an object facet to its associated class facet as a class attribute (i.e., an attribute whose value is the same for all instances of the class facet). For example, monovalued attributes name and sticker_price of CarModel are Type 1 in materialization CarModel--*Car (see Figure 3). Their value in object facet FiatRetro (Fiat-retro and 10.000, respectively) propagates as value of class attributes with the same name in class facet FiatRetro_Cars. 2. For users, Type 2 propagation concerns multivalued attributes of the more abstract class A. Their value for an instance of A determines the type, or domain, of instance attributes with the same name, monovalued or multivalued, in the associated class facet. Again, our semantics go through abstract objects and associated class facets. An example of the monovalued case is exhibited by attribute eng_size of CarModel. Its value {1200,1300} for the FiatRetro object facet is the domain of values for a monovalued instance attribute with the same name eng_size of the associated class facet FiatRetro_Cars. Thus, each FiatRetro car comes either with eng_size = 1200 or with eng_size = 1300. An example of the multivalued case is exhibited by attribute auto_sound of CarModel. Its value {tape, radio} indicates that each FiatRetro car comes with either tape, or radio, or both, or nothing at all as auto_sound. The associated class facet FiatRetro_Cars has a multivalued instance attribute auto_sound with the powerset :P({tape, radio}) as its type. 3. Type 3 propagation is more elaborate. It also concerns multivalued attributes of the more abstract class A, whose value is always a set of strings. Each element in the value of an attribute for object facet a generates a new instance attribute in the class facet associated with a. The type of generated attributes must be specified in the definition of the materialization. For example, attribute special_equip of CarModel propagates with Type 3
to Car. Its value {airbag, alarm, cruise_ctrl} for object FiatRetro generates three new monovalued instance attributes of type string, named airbag, alarm, and cruise_ctrl, for the associated class facet FiatRetro_Cars.
3
The TELOS data model
This section gives a general view of the main features of the T E L O S data model relevant to our formalization. More details about TELOS can be found in MBJK90. TELOS is actually supported by the ConceptBase system JJS96.
407 T E L O S is a language for representing knowledge about information systems. T E L O S knowledge bases are collections of propositions. Each proposition p is a three-tuple where from, label, and to denote the source, label, and destination of the proposition, respectively. These elements can be accessed through the functions From(p), Label(p), and To(p). TELOS propositions are either individuals or attributes. Individuals represent what are called objects (e.g., the individual book OOSC2ed) and classes (e.g., Book) in usual object models. While attributes represent binary relationships between individuals or other relationships. An example of an attribute is OOSC2ed, author, "B.
Meyer". Propositions can be classified in an arbitrary number of classification levels where each proposition is an instance of one or more generic propositions called classes. Classes that are themselves propositions must be in their turn instances of more generic classes, and so on. For example, OOSC2ed and OOSC2ed, author, 'B. Meyer' are instances of Book and Book, author, Person, respectively. The so-called w-classes can have instances along more than one level of classification. For example, Proposition has all propositions as instances and Class has all generic propositions as instances. The following example shows the use of the different features above. The TELL operation is used to add a new proposition in the knowledge base (i.e., create new objects in the terminology of usual object models) or to add new attributes to an already defined one. TELL TOKEN MTo93-#I In BorrowedDocument WITH author firitAutho~': "C. Marcos", secondAuthor: "M, Clha"; title 9 "A SDM approach for the Prototypmg of IS" borrowed : Yes; borrower "John" outDate : "05/06/97.9H" inDate :
END
"05/06/97:18H"
TELL CLASS Document IN Claus WITH attribute author Person, title: String, END TELL CLASS BorrowedDocument IsA Document, IN Class WITH attribute borrowed: String, borrower' Per=on; outDate Date; inDate. Date; END
Figure 4: TELOS definition of instances, classes, and attributes. Figure 4 shows, on the left side, the individual document MT-93-#l that is declared as an instance (via the IN clause) of the class BorrowedDocument defined on the right side of the figure as an instance of the metaclass Class and as a specialization of Document. The W I T H clause introduces the list of attributes. For example, the two first attributes of MT-93-#1, firstAuthor and secondAuthor, are two instances of the attribute class author. The attribute MT-
93-#I, firstAuthor, "C. Marcos" is an instance of Document, author, Person in exactly the same sense that MT-93-#l is an instance of Document. The third attribute of MT-93-#1 has no external label and it is an instance of the title class. Labels of such attributes are automatically generated by the system.
408 In Telos, a proposition may be an instance of more than one class (multiple classification). For instance, MT-93-#1 can be an instance of both classes MasterThesisDocument and RestrictedOocument which stands for a collection of documents that are not allowed to go out the library. M e t a - a t t r i b u t e s . The first-class status of attributes and the ability to define attributes and meta-attributes are very important in T E L O S . Figure 5 shows an example of meta-attributes which are needed to define common properties of the various resource classes. These meta-attributes are introduced through the metaclass ResourceClass. In this example, source, what, available, who, from, and until are meta-attributes which may be instantiated for ResourceClass instances. The class BorrowedOocument is declared now as an instance of ResourceClass on the right side of Figure 5 and its attribute borrower is an instance of the meta-attribute who. TELL CLASS ResourceClass WITH attribute source Class, what" Class, avadable. Class, who Class; from. Class, until Class; END
TELL CLASS BorrowedDocument IN ResourceClass WITH source author: person; who what borrower: Person; title. String; f~om available outDate: Date; borrowed, String, untd mDate: Date; END
Figure 5: Definition of meta-attributes. As another example of use of meta-attributes, Figure 6 gives the definition of the meta-attribute single that restricts its instances to (at most) a single value MBJK90. The right side of Figure 6 shows an example of use of the metaattribute single: we restrict the borrower of a BorrowedDocument to a single value by declaring it as an instance of single. The meta-attribute single is defined in the metaclass Class and it is inherited by BorrowedDocument by declaring BorrowedDocument as instance of Class. Note that by default a T E L O S attribute such as author: Person of Figure 5 can have several instances. If we want to restrict the attribute value, we have to use something like the meta-attribute single. Therefore, the declaration of attributes in T E L O S should not be confused with that of the usual object data models. TELL CLASS Class WITH attribute single' Class integrltyConstralnt smgle-Cnstr $ (V u/Classlsingle)(V p,q/Propositton) (p ,n u) A (q ,n u) A From(p) = From(q) =;~ (p = q) $ END
TELL CLASS BorrowedDocument IN ResourceClass, Class WITH .. who, single borrower Pecson; .. END
Figure 6: Definition of the single meta-attribute and its use. C o n s t r a i n t s ~ rules~ a n d m e t a f o r m u l a s . T E L O S supports an assertion sub-
409
language to specify integrity constraints and deductive rules. Constraints and rules are formulas that appear as attribute values of propositions. They specify the behavioral part of the objects to which they are attached. Constraints are assertions that control the information supplied by users, while deductive rules are assertions that enforce new facts. For example, the integrity constraint cl of the definition of Figure 7 ensures that the out of date for a borrowed document x must always be less than its return date. The constraint c2 ensures that a given document x cannot be borrowed by two persons at overlapping dates 2. The deductive rule states that once a person p borrows a certain document x, the system automatically derives the fact (x.borrowed = Yes), indicating that the document is actually borrowed. The "x/C" notation is read "x is an instance of C". TELL CLASS BorrowedDocument IN Class WITH
integrltyConstralnt
cl $ (V x/BorrowedDocument) (x outDate <~ x inDate)$; c2. $ (V x/BoTrowecTDocument)(Vp1, p2/Person) (x,borrower=pl) at t l A (x,borrower=p2) at t2 A (tl overlaps t2) => (pl = p2) $
deductiveRule 9 $ (V x/SorrowedDocument)(V p/Person) (x borrower = p) :=~ (x borrowed = Yes) $ END
Figure 7: Definition of constraints and deductive rules. In traditional modeling languages, a formula is defined for a given class to constrain the behavior of only the instances of this class. In T E L O S , the so-called metaformulas can be associated to a given metaclass to specify the behavior of both the instances of this metaclass and the instances of its instances. As an example, the constraint attached to the metaclass Oass on the left side of Figure 6 is a metaformula that manipulates p and q that are instances of instances of Class!single. To manipulate attributes and their values in definitions of formulas, we need the following functions where the time constraints are omitted MBJK90: 1. The dot function x.I evaluates to the set of values of the attributes of proposition x which belong to the attribute class labeled I. 2. The hat function x^l evaluates to the set of values of the attributes of proposition x with label I. 3. The bar function xI evaluates to the set of attribute propositions with source x which are instances of the attribute class labeled I. 4. The exclamation function xJl evaluates to the set of attribute propositions with source x and label I. 2TELOS also supports an explicit representation of time which is not presented in this paper (see MBJK90).
410
4
Formalizing the class level semantics of materialization
In this section we formalize the class level semantics of the materialization relationship by means of two metaclasses AbstractClass and ConcreteClass that represent, respectively, abstract and concrete classes in materialization hierarchies. TELL CLASS AbstractClas= In Class WITH attribute materializes: Conr END
TELL CLASS ConcreteClau In Class WITH attribute, single materOf AbsttactClass END
TELL CLASS AbstractClau WITH deductiveRule matetDedRule: $ (V A/AbstractClass)(V C/ConcreteClass) (C E A.materlahze=) ==P (C materOf = A) $ END
Figure 8: Definition of AbstractClass and ConcreteClass metaclasses. Figure 8 shows the definitions of the AbstractClass and ConcreteClass metaclasses. We declare AbstractClass as instance of the predefined metaclass Class. AbstractClass contains one meta-attribute whose label is materializes and destination is ConcreteClass. In the middle of Figure 8, we declare the metaclass ConcreteClass that plays the inverse role of AbstractClass. ConcreteClass contains one meta-attribute whose label is rnaterOf and destination is AbstractClass. The rnaterOf meta-attribute is constrained to be of a single value, meaning that a given concrete class has only one associated abstract class. On the right side of Figure 8, we add the deductive rule materDedRule to the AbstractClass metaclass to specify that once a given class A is declared as an abstract class which materializes in a given concrete class C, the system automatically induces the fact (C.rnaterOf = A) which means that C is a materialization of the abstract class A. A similar deductive rule can be associated with the ConcreteClass metaclass to play the dual role. 4.1
Definition of the materialization
characteristics
Materialization characteristics are formalized as attributes of the meta-attribute materializes. To be able to attach properties to materializes, we have to declare this later as a metaclass as shown in Figure 9. In Figure 9, we apply the "!" symbol to AbstractClass to access the attribute materializes itself. The figure shows the following characteristics: cardinality denotes the cardinality of an abstract class regarding a concrete class. The trivial associated constraint minrnaxCard states that the minimal cardinality is always less than the maximal one. The remaining attributes labeled inbAttrT1, inhAttrT2, and inhAttrT3 specify propagation modes for attributes of the abstract class to the corresponding concrete class. Definitions of their destinations (i.e., domains) are given on the right side of the figure: 1. Attribute-lDef is the name of an attribute propagating with Type 1;
411
TELL CLASS AbstractClais!materlahzes In Class, Attnbute WITH attribute cardlnallty CardType, mhAttrTl: Attribute-lDef, InhAttrT2. Attribute-2Def, inbAttrT3 Attribute-3Def TELL CLASS CardType In Class WITH attribute rain: Integer; max. Integer integntyConstramt minmaxCard. S(V c/CardType) (c min <. r max)$ END
TELL CLASS Attribute-lDef In String END TELL CLASS Attribute-2Def In Class WITH attribute attrName String, derivAttr: String { * mono or multi * } END TELL CLASS Attr*bute-3Def In Clan WITH attribute attrName. String, genAttrType: TypeDef; END TELL CLASS TypeDef In Class WITH END
Figure 9: Definition of the materialization characteristics. 2. Attribute-2Def gives, for an attribute propagating with Type 2, its name and the kind derivedAttr (rnonovalued or rnultivalued) of the derived instance attribute 3; 3. Attribute-3Def gives the name of an attribute propagating with Type 3 and the value type genAttrType (TypeDef). Note that the meta-attribute inhAttrT1 (resp., inhAttrT2 and inhAttrT3) can be instantiated in application with as many Type 1 (resp., Type 2 and Type 3) attributes as needed. 4.2
Example
of use of generic semantics
As an example, Figure 10 shows how the CarModel--*Car materialization is formalized by invoking the generic semantics. In Figure 10 (a), the classes CarMode and Car are created as ordinary classes, independently of the notion of materialization. To take into account the materialization relationship between fiarMode and Car, we declare in Figure 10 (b) fiarModel as an instance of Abstractfilass and Car as an instance of ConcreteClass. During the creation of CarModel we instantiate the meta-attribute materializes of AbstractClass by rnaterializesCar. Note that there is no need to instantiate the meta-attribute rnaterOf of ConcreteClass during the creation of Car. This will be automatically achieved by the deductive rule rnaterDedRule of Figure 8 (b). In the attribute CarModelknaterializesCar we specify that: the cardinality of CarModel regarding Car is 0:N; name and sticker.price propagate with Type 1; #doors and eng_size both propagate with Type 2 and each produces a rnonovalued instance attribute, while auto_sound produces, also with Type 2, a rnultivalued instance attribute; special_equip generates, with Type 3, new instance attributes of type String. Generic semantics of materialization given above also hold for abstract classes that materialize in m o r e than one concrete class such as a hierarchy of materializations CI*--A--*C2. In such a case, we create A as an instance of AbSThe {* ...*} notation denotes a TELOS comment.
412
TELL CLASS CarModel In Class WITH attribute name: String, sticker.price Integer; ~doors: Integer; eng-slze Integer; auto.sound: String. speclal.equlp" String END
TELL CLASS Car In Class WITH attribute manuf ~late 9 Date; serial# Integer, owner Strlnlg END
(a) TELL CLASS CarModel In Ab~tractClass WITH materlahze= materiallzesCar Car END TELL CLASS Car In ConcreteClass WITH END TELL CLASS CarModel!materlalizesCar In Class WITH cardlnality. 0.N mhAttr=bTl T l l : name, T12" stlcker.prlce inhAttribT2 T21: T2Doors; T22 T2EngSize, T23. T2AutoSound inhAttrlbT3, T31: T3SpeclalEqulp END
TELL TOKEN name In Attribute-lDef WITH END TELL TOKEN sticker.price In Attribute-lDef WITH END TELL TOKEN T2Doors In Attnbute-2Def WITH attrName. "~doors", derwAttr. "mono" END TELL TOKEN T2EngSize In Attnbute-2Def WITH attrName. "eng.size"; derlvAttr "mono" END TELL TOKEN T2AutoSound In Attribute-2Def WITH attrName' "auto-sound" ; derlvAttr: "multi" END TELL TOKEN T3Spec=alEquipIn Attnbute*3Def WITH attrName' %pecial.equip" ; genAttrType: String, END TELL CLASS String IN TypeDef END
TELL TOKEN 0.N In CardType WITH mln 0; max' 100 { * 100 denotes here the ~ (N) symbol ~} END
(b) Figure 10: Materialization CarModel---*Car. stractClass and instantiate the attribute materializes into two attributes materializesCl and materializesC2 with destinations C1 and C2, respectively. Then, we declare A!materializesC1 and A!materializesC2 as instances of the metaclass Class to capture different characteristics of materializations A---*C1 and A--*C2, respectively. 4.3
Constraints
related
to inheritance
attributes
This section defines two constraints related to inheritance attributes. The first one expresses the membership of inheritance attributes to abstract classes and the second states that Type 2 attributes must be multivalued. Inheritance attributes are m e m b e r s of abstract classes. All inheritance attributes appearing in the definition of the meta-attribute AbstractClass!materializes must be attributes of the more abstract class. For instance, the Type 1 attributes (name, sticker_price), the Type 2 attributes (#doors, eng_size, and auto_sound), and the Type 3 attribute (special_equip) of the Figure 10 (b) must be attributes of the abstract class CarModel. Figure 11 shows, respectively, the constraints T1Attr_Cnstr, T2Attr_Cnstr, and T3Attr_Cnstr that express the membership of the inheritance attributes to the abstract class of a materialization relationship.
413
TELL CLASS AbstractClass!materializes TELL CLASS AbstractClass!materiahzes TELL CLASS AbstractClasslmateriallzes WITH mtegrltyConstramt WITH mtegr=tyConstraint WITH integrltyConstramt T1Att~.Cnstr T2Attr.Cnstr' T3Attr.Cnstr: $ (V M/AbstractClasslmateriahzes) $ (V M/AbstractClass!materiahzes) $ (V M/AbstractClasslmatetlahzes) (Y A/AbstractClass) From(M)=A =r (V A/AbstractClass) From(M)=A ~P (Y A/AbstractClass) From(M)=A =~ (V T1/Attribute-lDef)(V S1/Str,ng) (V T2/Attnbute-2Def)(V S2/Strmg) (V T$/Attrlbute-SDef)(V $3/Strlng) (T1 E M.mhAttrT1 A To(T1)=S1 (T2 E M inbAttrT2) A (S2 = T2 attrName) (T3 E M inhAttrTS) A ($3 = T3 attrName) =r (5 p/Proposition) ==~ (:1 p/Propositlon) :=~ (:1 p/Proposition) (p E A I attrlbu~) (p E A I attr,bute) A Label(p)=S1 $ (p E A ~ attnbute) A Label(p)=S2 $ A Label(p)=S3 $ END END END
Figure 11: Membership metaconstraints of inheritance attributes.
T y p e 2 attributes are multivalued. Further the constraints related to membership of the inheritance attributes to the abstract classes, Type 2 attributes (e.g., #doors, eng_size, and auto_sound of Figure 10 (b)) must be multivalued: they must have more than one value at the instance level (e.g., in FiatRetro of Figure 3, #doors has two values 1200 and 1300). TELL CLASS AbstractClass WITH integrltyConstralnt T2attr .are.multlvalued: $ (Y A/AbstractClass)(Y M/AbstractClass!materiahzes) From(M)=A =~ (V T2/Attribute-2Def)(V S2/String) (T2 E M.inhAttrT2) A ($2 = T2.attrName) =r (3 p/Proposlt,on) (p E A I attribute) A Labei(p)=S2 ~> (p in Class!multNalued) $ END
TELL CLASS ClassWITH attribute multivalued Clals integrltyConstramt muitlvalued.Cnst r. $ (V attm/Classlmultivalued)( =~ p,q/Proposltlon) (p In attm) A (q in attm) A From(p) : From(q) =~ (p ~ q) $
END
Figure 12: Metaconstraint related to values of Type 2 inheritance attributes.
The left side of Figure 12 shows the definition of the constraint ensuring the above requirements, The Type 2 attributes are declared as instances of the meta-attribute Class!multivalued we define on the right side of Figure 12. This figure shows that the source and the destination of the attribute labeled multivalued are of type Class. The constraint associated with multivalued states that for every instance attm of Class!multivalued, there are at least two distinct instances p and q with common source4. According to the constraint of Figure 12, Figure 10(a) must be revised: the Type 2 attributes (~doors, eng_size, and auto_sound) must have been declared as instances of the meta-attribute multivalued. 4.4
Cardinality
constraints
In this section we formalize the cardinality constraints of materialization.
Definition of the 0:N and 1:1 cardinalities. Figure 13 shows definitions of two constraints card0_NCnstr and cardl_lCnstr which formalize, respectively, 4The meta-attribute multivalued is defined in the same spirit as the meta-attribute single (see Figure 6).
414 the cardinalities 0:N associated with abstract classes and 1:1 associated with concrete classes. In Figure 13, we manipulate the abstract and concrete objects (e.g., a and c) that are, respectively, instances of AbstractObject and Concrete0bject classes we define in the next section. The last implication of the constraint card0_NCnstr is always evaluated to TRUE, meaning that we tolerate both the existence and the absence of a concrete instance c for a given abstract object a. The cardl_lCnstr definition is composed of two parts: the existence part which states that for each concrete object there is an associated abstract object and the uniqueness part stating that there is one and only one associated abstract object. TELL CLASS A~tractClass!materlallze~ WiTH TELL CLASS ConcreteClasslmaterOf WITH integtltyConstramt integtltyConst raint card0J~fCnstr, cardllCnstr ' $ (V M/AbstractClasslmaterlahze~) $ (V M/ConcreteClasslmaterO()(V A/AbstractClass)(V C/ConcreteClass) (V A/AbstractClass)(V C/ConcreteClass) From(M)=C A To(M)=A A (M cardinality = 1.1) =~ From(M)=A A To(M)=C A (M,cardinality = 0.N) (V c/ConcreteObject) (e m C) :r =~ (Y a/Ab~tractObject) (a in A) =~ (=1 a/AbstractObject) (a ll1 A) A (a E c materOf) { * existence * } A (:~ c/ConcreteObject) (r In C) A (c E a,materlallzes) (V al, a2/AbstractObiect) (al m A) A (al E e materOf) A (a2 in A) A =~ (TRUE) (a2 E r =~- (al = a2)) { * unlquenen * } $ $ END END TELL TOKEN 1.1 In CardType WITH TELL TOKEN 0.N In CardType WITH ram: 1. mln: 0, max, 100 max: 1 {* 100 denotes here the oo (N) symbole * } ENO END
Figure 13: Definition of the 0:N and 1:1 cardinality constraints. Definition of t h e Cmin, Cma~ cardinality. As for the Cmin, C,na, cardinality (e.g., 3,6), which may be associated at the side of abstract classes, there is no way, in Telos, for its formalization. To deal with this category of cardinality, we define a new predicate length(X, Class, Assertion, Integer) with the following meaning: the last argument is the number of elements X, instances of the second argument, satisfying the assertion given in the third argument. Formally, we can write:
I Length(x, C, P, N) = # {x/C Holds(P(x))}= N The predicate Holds is true whenever its argument is an assertion that follows from the knowledge base MBJK90. The "#" symbol is applied to a given set and denotes the number of its elements. As a result of this new predicate, the constraint related to the Cmin, Crnax cardinality will be as shown in Figure 14. TELL CLASS Ab~tractClas~!materlallzes WITH integtityConstralnt cardMm.MaxCnstr, $ (V M/Ab~tractClasslmaterlahzes) (V A/AbstractCla~.s)(V C/ConcreteClass) From( M)=A A To( M)=C A 1 ~ M,cardlnallty, mm < M cardmahty, max A M.cardmalJty max > 2 =~> (Y a/AbstractObject) (a In A) A (a in AbstractOb.iect) =P Length(c, ConcreteObject, (c ~n C) A (c E a matermlizes), K) A (Min.Max,min < K <Min.Max.max) $ END
Figure 14: Definition of the constraint related to the cardinality Cmin,
Cma=.
415
5
Formalizing the instance level s e m a n t i c s of materialization
This section describes the instance level semantics of materialization by means of two classes AbstractObject and ConcreteObject that represent, respectively, abstract objects and concrete objects, instances of instances of the metaclasses AbstractClass and ConcreteClass, respectively. TELL CLASS AbstractObject In Clau WITH attribute r Class matenahzes ConcreteObject END
TELL CLASS ConcreteObject In Class WITH attribute, s~ngle materOf AbstractObject END
TELL CLASS AbltractObject WITH deductiveRule materDedRule: $ (V a/AbstractObject)(V c/ConcreteObject) (c E a.matenahzes) =~ (c rnaterOf = a) $ END
Figure 15: Definitions of AbstractObject and ConcreteObject classes. The definitions of the classes AbstractObject and ConcreteObject are depicted in Figure 15. On its left side, the figure shows that each abstract object has an associated class facet, denoted classFacet. An abstract object has also an attribute materializes whose domain is ConcreteObject and each concrete object has an attribute rnaterOf whose domain is AbstractObject. For reason of uniformity, the same names materializes and rnaterOf axe used both for the description of the class level and the instance level of materialization semantics. The attribute rnaterOf is declared as an instance of single to assert that a concrete object is materialization of only one abstract object. Finally, as in Figure 8, we give on the right side of Figure 15 a deductive rule that automatically derives from the fact (c 9 a.materializes) another fact (c.rnaterOf = a) indicating that c is a materialization of a. 5.1
Constraints
related to the abstract
objects
Figure 16 shows two constraints concerning the abstract objects. On the left side we define the abstractObj_Cnstr constraint which expresses that instances of instances of AbstractClass must be instances of AbstractObject. On the right side, the ObjClassFacet_Cnstr constraint means that each abstract object a has one and only one (denoted by the "3!" symbole) associated class facet regarding a given materialization. The constraint also shows that the class facet must be a subclass of the concrete class C that is a materialization of the abstract class A, the class of a. The ObjClassFacet_Cnstr constraint implicitly means that, each time the user creates an abstract instance a, he/she also must create explicitly its associated class facet to ensure the constraint. We think that it would be more reasonable and more natural to implicitly generate the class facet of a given abstract object. For this, we propose to extend the definition of the deductive rule. The actual
416
TELL CLASS AbstractClass WiTH integrltyConstraint ab=tractObjoCnltr. $ (V A/AbstractClass)(V a/Class) (a in A) =~ (a in AbstractObject) $ END
TELL CLASS AbstractClass WITH integrityConstramt ObjClauFacet.Cnstr: $ (V A/AbetractClass)(Y C/ConcreteClass) {V a/Ab~tractObjact) (a m A) A (C E A.rnateriahzes) =~ (:11 Cf/Clar~) (Cf I$A C) A (Cf E a classFacet) $ END
Figure 16: Constraints associated with the abstract objects.
definition of a deductive rule allows only the possibility to derive facts that are logical expressions. The extension we propose consists of the specification, on the right side of a deductive rule, of some operations we wish to be executed at run time as for the rule-based system MARVEL KFP88. Such an extended rule will be of the pattern: < preconditions > ~ < usual facts > and < operations > where < usual facts > is the ordinary part we find on the right side of a usual deductive rule and < operations > stands for operations we wish to be automatically executed by the system when the part < preconditions > is satisfied. The ObjClassFaceLCnstr constraint on the right side of Figure 16 becomes the following extended deductive rule:
TELL CLASS AbstractClasi WITH deductlveRule Ob/ClassFacet Ext .Cnstr: $ (V A/AbstractCl~ss)(V C/ConcreteClau) (V a/AbstractObject) (a in A) A (C E A.materlalizes) =~ TELL CLASS Cf in Class, isA C WITH END A (Cf E a.classFacet) $ END
This deductive rule means that for each abstract object a, instance of an abstract class A which materializes in C, the system automatically generates a class Cf as an instance of Class and as a subclass of C. The system induces then the fact (Cf E a.classFacet) meaning that Cf is a potential class facet of a. 5.2
Constraints
related
to the concrete
objects
Figure 17 shows two constraints regarding the concrete objects. On the left side, the concObj_Cnstr constraint expresses that instances of instances of ConcreteClass must be instances of ConcreteObject. On the right side, the concObjClassFacet_Cnstr constraint expresses that all concrete objects c, instances of a given concrete class C, and materializing a given abstract object a, must be instances of the class facet associated with a. For example, Nico's Fiat that is instance of Car and that materializes FiatRetro must be instance of FiatRetro_Cars, the class facet associated with FiatRetro (see Figure 2).
417
TELL CLASS ConcreteClass WITH
TELL CLASS ConcreteClass WITH
integrltyConstraint
integrityCon=traint
concObjClassFacet.Cnstr: $ (V A/Al~tractClass)(V C/ConcreteClass) (C (~ A.materlalize=) =r (V c/ConcreteObject) (V a/AbstractObject) (a in A) A (Cf E a.classFacet) A (Cf isA C) A (c in C) A (c E a materializes) ==~ (c In Cf)
concObj.Cnst r $ (Y C/ConcreteClass)(V c/Class) (c in C) =~ (c in ConcreteObject) $
END
$
END
Figure 17: Constraints associated with the concrete objects.
5.3
Constraints
related
to attribute
propagation
P r o p a g a t i o n o f T y p e 1 a t t r i b u t e s . Figure 18 shows, on its left side, the constraint T1AttrlnClassFacet_Cnstr which imposes the Type 1 attributes of an abstract class A to be attributes of each class facet associated with each instance a of A. The right side of Figure 18 shows the constraint T1AttrAreClassAttr_Cnstr which states that Type 1 attributes are class attributes 5 in class facets: for a given Type 1 attribute T1, if the value of T1 in a given abstract object a is v then all the instances of the class facet associated with a will come with the same value v. Another alternative to deal with Type 1 attributes is the use of delegation mechanism Lie86 that would permit concrete instances to access directly the Type 1 attribute values in the abstract instance a, without storing it redundantly in class facets. Unfortunately, TELOS does not provide facilities for using the delegation mechanism.
TELL CLASS AbstractClas= WITH integrltyCon$t ralnt
TIAtt rlnClassFacet.Cnit r' $ (V M/AbstractClasslmaterlallze$) (V A/AbstractClass) (Y C/ConcreteClass)(V T1/Att~,bute-lDef) (Y S1/Strmg) (V D/Domain) (C E A.M) A (T1 E M.,nhAttrTt) A To(T1)=SI =~ (3 p/Proposltmn) (p E A I attribute) A Label(p)=Sl A To(p)=D =r (V a/AbstractObject) (V Cf/Class) (a in A) A (Cf E a.classFacet) A (Cf ,sA C) ::~ (3 q/Propos,tlon) (q E Cf { attribute) A Label(q)=S1 A To(q)=D $
END
TELL CLASS AbstractClas$ WITH integrltyConstralnt
T 1Att rAreClassAttr .Cnstr: $ (Y M/AbstractClasslmatenahzes) (V A/AbstractCla~) (V C/ConcreteCla~)(V T1/Attribute-lDef) (Y S1/String) (V D/Domaln) (C E A.M) A (T1 E M.inhAttrT1) A To(T1)=Sl =r (3 p/Propo~,tion) (p E A I attribute) A Label(p)=S1 A To(p)=D ~> (Y a/AbitractObject) (Y Cf/Class) (a in A) A (Cf E a classFacet) A (Cf isA C) =~ (V q/Proposibon) (q E Cf I attribute) A Label(q)=S1 A To(q)=D ---> (V u,v,c/Proposltlon) (c in Cf) A (U In q) A From(u)=r A To(u)=v ==V (v = a.S1) $ END
Figure 18: Constraints associated with Type 1 attribute propagation.
P r o p a g a t i o n o f T y p e 2 a t t r i b u t e s . The constraint T2AttrlnClassFacet_Cnstr of the left side of Figure 19 imposes the Type 2 attributes of an abstract class A to be also attributes of each class facet associated with each instance a of A. The domain of the Type 2 attributes in class facets is the same as in the abstract class, but it will be, next, restricted to only a set of fixed values. The 5we mean here the usual sense of "class attributes" as opposed to "instance attributes".
418 TELL CLASS AbstractClass WITH
mtegrltyConstralnt T2Attrln ClassFacet.Cnstr:
TELL CLASS AbstrattClass WITH
$ (Y M/AbstractClas$!materiahze=) (y A/Ab=tractClas$) (Y C/ConcreteClass)(Y T2/Attrlbute-2Def) (V S2/Strlng)(V D/Domain) (C E A.M) A (T2 E M.inhAttrT2) ^ ($2 = T2.attrName)
mtegntyConstraint T2ValuePropag.Cnstr: $ (V M/AbstractClass=mate~=allzes)(V A/Al~tractClass) (V C/ConcreteCiar~) (Y T2/Attribute-2Def)(Y S2/String)
(3 p/Proposition) (P E A I attribute) A Label(p)=S2 A To(p)=D ==~ (Y a/AbstractObject)(V Cf/Cla~) (a In A) A (Cf E a,classFatet) A (Cf i=A C) => (:1 q/Proposition) (q E C{ I attribute) A Label(q)=S2 A To(q)=D $ END
(C E A M) A (T2 E M.mbAttrT2) A ($2 = T2.attrName) =~ (Y a/AbstractObject)(V Cf/Class)(V q/Proposltion) (a in A) A (Cf E a classFacet) A (Cf I$A C) A (q E CF J attribute) A Label(q)=S2 ==F (T2 derlvAttr = "mono") ==P (q in Classlsmgle} A ((Y u.v,c/Prop~lt,on) (c m Of) A (= ,. q) A From(u)=c A To(u)=v =~ (v E a.s2) ) A (T2, derlvAttr = "multi") ==~ (Y u,v,r (r In Cf) A (u in q) A From(u}=r A To(u)=v ==> ( v E aS2)$
END
Figure 19: Constraints associated with Type 2 attribute propagation.
constraint responsible of this restriction is T2ValuePropag_Cnstr as given on the right side of Figure 19. This constraint expresses that for each concrete instance c, instance of a class facet Cf, the value of its Type 2 attribute S2 must belong to the set of values of $2 in the abstract object a that materializes in c. If $2 is monovalued, then $2 must have only one value in c, otherwise $2 can have several values. For instance, the Type 2 attribute "#doors:Integer" of CarModel must be also an attribute of the class facet FiatRetro_Cars associated with FiatRetro, instance of CarModel. The domain of #doors in FiatRetro_Cars is also Integer, but its value in Nico's Fiat, instance of FiatRetro_Cars, must be restricted to either 3 or 5: the two possible values fixed by the abstract instance FiatRetro. P r o p a g a t i o n of T y p e 3 attributes. Figure 20 shows the T3ValuePropag_Cnstr constraint that formalizes the propagation of Type 3 attributes. It shows that each value V3 of a given Type 3 attribute in an abstract object a is a new attribute of the class facet Cf associated with a. The domain of V3 in Cf is D which a user can supply in advance by using the attribute genAttrType associated with the definition of Type 3 attribute (Attribute-3Def).
TELL CLASS Al~t~actClass WITH
integntyConstraint T3ValuePropag.Cnst r $ (Y M/Ab~tractCla=slmaterlahzes)(Y A/AbstractClass) (Y C/ConcreteClass)(Y T3/Attr, bute-3Def) (V S3/Strlng)(V D/TypeD#) (C E A M) A (T3 • M mhAttrT3) A ($3 = T3.attrName) A (T3.genAttrType = D) =~ (3 p/Proposition) (p E A I attribute) A Label(p)=S3 ==(V a/AbstractObjec~)(V Cf/Class) (V V3/Class) (a ,n A) A (Cf E a.clactFacet) A (V3 E a $3) =~ (=I q/Propositlon) (q E Cf I attribute) A Label(q)=V3 A To(q)=D
$
END
Figure 20: Constraints associated with Type 3 attribute propagation.
419
6
Conclusion
This paper has presented a formalization of materialization, a new and ubiquitous abstraction pattern relating a class of abstract categories and a class of more concrete objects. Materialization allows the definition of new and powerful attribute propagation (or inheritance) mechanisms from the abstract class to the concrete class. Our formalization was carried out using the metaclass approach of T E L O S . Two metaclasses AbstractClass and ConcreteClass were built as templates to capture the semantics of materialization at the class level and two additional classes Abstract0bject and Concrete0bject were defined to capture the semantics of materialization at the instance level. The metaclasses AbstractClass and ConcreteClass define the meta-attributes materializes and rnaterOf, respectively. Thanks to the class status of T E L O S attributes, the meta-attributes materializes and rnater0f are declared as ordinary metaclasses to which we attached all characteristics of materialization relationship. Various constraints have been uniformly defined at the levels of classes, metaclasses, and attributes to ensure the semantics of materialization at both the class and the instance levels in a coordinated manner. The metaclass approach allowed us to define the semantics of materialization as a unit which is independent of any implementation consideration. Consequently, our formalization can be used to implement materialization in various systems. Although our formalization is more suitable for implementation systems that support metaclasses, it can be also used to assist an implementation in non metaclass-based systems. Furthermore, our approach can be followed to formalize other new generic relationships. This work had also demonstrated the power of the T E L O S language to express the requirements related to the materialization semantics. To fully formalize materialization, we proposed two suggestions to extend T E L O S . One consisted of the definition of a new predicate required to formalize the cardinality Cmin, Cma=. The second one consisted of the extension of the deductive rule definition for specifying some operations we would wish to be automatically executed by the system when given preconditions are satisfied. Our work has several continuations. An interesting one deals with the formalization of the common semantics of a large repository of new generic relationships (e.g., Member-Of MPS95, Role-Of WDJS94, Part-Of KP97, Ownership HPYG95). Another continuation will be to start from our formalization and try to realize an effective implementation in a target system (e.g., ConceptBase) in such manner that it will be possible to evaluate this system with regard to its power in the implementation of materialization semantics. Acknowledgements.
I wish to thank all the colleagues in the Y E R O O S
420
project for their critical comments on an earlier draft of this article and particularly M. Guy Fokou. I am also grateful to Christoph Quix (RWTH Technical University of Aachen) for many useful discussions about TELOS and ConceptBase. Finally, I thank Professor Matthias Jarke (RWTH Technical University of Aachen) for the permission to use the ConceptBase system.
References Dah97
M. Dahchour. Formalizing materialization in the TELOS data model. Technical Report TR-97/28, IAG-QANT, Universit@ catholique de Louvain, Belgium, November 1997.
DPZ96
M. Dahchour, A. Pirotte, and E. Zims Metaclass implementation of materialization. Technical Report YEROOS TR-96/06, January 1996. Submitted for publication.
DPZ97
M. Dahchour, A. Pirotte, and E. Zims Metaclass implementations of generic relationships. Technical Report YEROOS TR-97/25, 1997. Submitted for publication.
GSR96
G. Gottlob, M. Schrefl, and B. RSck. Extending object-oriented systems with roles. A CM Trans. on O~ce Information Systems, 14(3):268-296, 1996.
HGPK94
M. Halper, J. Geller, Y. Pert, and W. Klas. Integrating a part relationship into an open OODB system using metaclasses. In N.R. Adam, B.K. Bhargava, and Y. Yesha, editors, Proc. of the 3rd Int. Conf. on Information and Knowledge Management, CIKM'94, pages 10-17, Gaithersburg, Maryland, November 1994. ACM Press.
HPYG95
M. Halper, Y. Perl, O. Yang, and J. Geller. Modeling business applications with the OODB ownership Relationship. In Proc. of the 3rd Int. Conf. on AI Applications on Wall St., pages 2-10, June 1995.
JJS96
M. Jarke, M.A. Jeusfeld, and M. Staudt. ConceptBase V~.I User Manual. 1996.
KFP88
G. E. Kaiser, P. H. Feiler, and S. S. Popovich. Intelligent assistance for software development and maintenance. IEEE Software, 5(3):4049, 1988.
KP97
M. Kolp and A. Pirotte. An aggregation model and its C++ implementation. In Proc. of the ~th Int. Conf. on Object-Oriented Information Systems, Brisbane, Australia, 1997. To appear.
421
KS95
W. Klas and M. Schrefl. Metaclasses and their application. LNCS 943. Springer-Verlag, 1995.
Lie86
H. Lieberman. Using prototypical objects to implement shared behavior in object oriented systems. In N.K. Meyrowitz, editor, Proc. of the Conf. on Object-Oriented Programming Systems, Languages and Applications, OOPSLA '86, pages 214-223, Portland, Oregon, November 1986. ACM SIGPLAN Notices 21(11), 1986.
MBJK90
J. Mylopoulos, A. Borgida, M. Jarke, and M. Koubarakis. Telos: representing knowledge about informations systems. ACM Trans. on O~ce Information Systems, 8(4):325-362, 1990.
IMP93
R. Motschnig-Pitrik. The semantics of parts versus aggregates in data/knowledge modelling. In C. Rolland, F. Bodart, and C. Cauvet, editors, Proc. of the 5th Int. Conf. on Advanced Information Systems Engineering, CAiSE'93, LNCS 685, pages 352-373, Paris, France, June 1993. Springer-Verlag.
MPK96
R. Motschnig-Pitrik and J. KaasboU. Part-whole relationship categories and their application in object-oriented analysis. In Proc. of the 5th Int. Conf. on Information System Development, ISD'96, September 1996.
MPS95
R. Motschnig-Pitrik and V.C. Storey. Modelling of set membership: The notion and the issues. Data FJ Knowledge Engineering, 16(2):147-185, 1995.
PZMY94
A. Pirotte, E. Zims D. Massart, and T. Yakusheva. Materialization: a powerful and ubiquitous abstraction pattern. In J. Bocca, M. Jarke, and C. Zaniolo, editors, Proc. of the 20th Int. Conf. on Very Large Databases, VLDB'9~, pages 630-641, Santiago, Chile, 1994. Morgan Kaufmann.
WDJS941 R. Wieringa, W. De Jonge, and P. Spruit. Roles and dynamic subclasses: a modal logic approach. In M. Tokoro and R. Pareschi, editors, Proc. of the 8th European Conf. on Object-Oriented Programming, ECOOP'94, LNCS 821, pages 32-59, Bologna, Italy, July 1994. Springer-Verlag.
Aligning Legacy Information Systems to Business Processes Panos Kardasis & Peri Loucopoulos Department of Computation Department U.M.I.S.T. P.O. Box 88, M60 1QD Manchester, UK {kardasis | pl} @co.umist.ac.uk Abstract Recent years have experienced a growth in demand for re-engineering legacy information systems. The complexity of a development endeavour leading to migration of a legacy system stresses the need for a systematic supporting approach. We argue in this paper that such an approach should facilitate (a) documentation of dependencies between business processes and supporting IS in a way that changes in the business level are reflected on system specifications and (b) quantification of effects of business changes on the associated migration from legacy systems so that alternative solutions can be systematically evaluated and the optimal solution can be chosen. In this paper we put forward an approach to meeting these two objectives based on the confluence of two technologies: Enterprise Knowledge Modelling and Knowledge Discovery in Data. The approach is demonstrated using examples from a project involving a banking application.
1
Introduction
Over the past decade, continuous challenges have been made to traditional business practices. Many institutions, companies and virtually all industries have been forced into reactive patterns of change in order to remain competitive. This effort has witnessed the disabling effects that the build-up of legacy systems has on such change. Immense size and criticality in the overall business operation, use of inappropriate and obsolete hardware, poor database services, lack of interface among applications and presence of unstructured and undocumented patches are only some of the typical characteristics of legacy systems [Brodie and Stonebraker 1996]. As an effect, migration from legacy environments is certainly not a trivial process, while it may become extremely expensive and time consuming. Projects aiming at the replacement of legacy Information Systems (IS) by totally new ones tend to fail for a number of reasons: •
Specifications for the legacy systems rarely exist, relevant documentation is outof-date or lost, and thus, the only source of information is the system code.
•
Too much effort is spent on the analysis phase, which may never end, either because of the complexity of the system, or because of the ineffectiveness of the analysis approach.
•
The replacement of the IS raises the need for business changes, which are rarely welcome. The fear of change in working practices, techniques and enabling
B. Pernici and C. Thanos (Eds.): CAiSE’98, LNCS 1413, pp. 25-39, 1998. Copyright Springer-Verlag Berlin Heidelberg 1998
26
Panos Kardasis and Peri Loucopoulos
technologies contribute to enormous resistance. The large migration projects become even larger as everybody in the company is eventually found to be either involved or affected. This makes the situation uncontrollable and the project vulnerable to termination. The above observations stress the need for a systematic approach to assist system migration. Such an approach should support: •
Understanding of the enterprise in terms of its operations and resources, in order to provide a solid background for analysing the system, and for assisting the coordination of business changes dictated by the migration project.
•
Documentation of dependencies between business processes and supporting IS in a way that changes in the business level are reflected on system specifications, and vice versa.
•
Quantification of effects of business changes on the associated migration from legacy systems so that system migration strategies can be systematically evaluated and the optimal solution can be chosen.
In this paper we put forward an approach to meeting these objectives based on the confluence of two technologies: Enterprise Knowledge Modelling and Knowledge Discovery in Data. The term ‘enterprise knowledge modelling’ refers to the set of techniques for describing the structure and business processes of an enterprise, its missions and objectives together with the way that these objectives may be operationalised onto system components [Bubenko, Rolland, et al 1994; Loucopoulos 1994; Loucopoulos and Katsouli 1992; Loucopoulos and Kavakli 1995; Rolland and Grosz 1994]. The term ‘ knowledge discovery in data’ refers to correlations between data variables, identification of rules, and classifications implicitly contained in large amounts of corporate data [Matheus, Chan, et al 1993; Yoon and Kerschberg 1993]. Enterprise knowledge modelling provides the basis for developing models of current business processes and objectives for change, including changes to the business goals and business rules. Knowledge Discovery in Data is used for investigating the behaviour of the legacy IS in terms of its operational data and the way that such data is presently being used within the chosen business processes; it can also be used for identifying behavioural patterns which may give rise to new business processes [Dhar and Tuzhilin 1993]. The approach advocated in this paper is underpinned by three key activities: 1. Modelling of enterprise objectives, rules, and processes – for describing both the AS-IS and the TO-BE situations. 2. Analysis of legacy IS – for discovering the actual behaviour of the system against a set of defined business criteria.
Aligning Legacy Information Systems to Business Processes
27
3. Matching business knowledge models to results from analysis of legacy IS – for identifying the necessary changes to the IS (primarily) but also potential changes to the business processes themselves. These three activities represent the backbone of the paper. The discussion is also based on practical grounds by considering an industrial application within the banking domain. Following a short introduction of the business application and of the background modelling approach (section 2), the paper discusses the modelling activity (section 3), the knowledge discovery activity (section 4) and the integration activity (section 5), using examples from the banking application in order to demonstrate the issues being discussed. Finally, the paper concludes with a number of observations (section 6). 2
The Banking Application
The examples presented in this paper are part of a large project aiming at enabling customer profiling in the banking sector. The part of the application in which this paper is confined is one particular business area namely the marketing of loans, hire purchase agreements, and preference accounts through a dense network of local sales representatives and external agents. The critical business functions deal with: • Targeting of the customer base and contacting customer target groups through massive mailings • Contacting individual customers in order to make offers and agreements • Evaluating customer applications by considering both the profitability and the risk of the proposed agreements These main functions are supported currently by a legacy IS, an abstract view of which is shown in the diagram of Figure 1. The first process is supported by the ‘TCS’ system (Figure 1) which alerts sales representatives about customers that need to be called each time. Offers to existing customers are generally aligned to previous interaction between the customer and the bank, the history of which is provided accordingly by the system. However, the decision about the type of product to be offered and the agreement terms are left to the sales representative himself. The second process is supported by the component shown in Figure 1 as ‘Application Systems’. This component is composed of many application programs that collectively deal with the scoring of customer requests according to customers' personal details, behaviour from previous transactions with the bank and references from external agencies. Finally, the ‘Marketing Database’ brings together the contents of all other database systems in order to facilitate decision making for massive mailing campaigns. The majority of the systems supporting the current situation were developed and put to work within the past three decades or were occasionally appended to the existing systems in an undocumented, ad-hoc manner.
28
Panos Kardasis and Peri Loucopoulos
&XVWRPHU5HFRUGV
7&6 6\VWHP
3URVSHFW 'DWD
&XVWRPHU
0DLOLQJ $JHQF\
'DWD
3URVSHFW 'DWD
$QDO\VLV 7RROV
&XVWRPHU 5HFRUGV
0RQWKO\
HQTXLULHV HWF
5HFRUGV
3URVSHFW 'DWDEDVH 8SGDWHV
5HVSRQVHV
&XVWRPHU
$SSOLFDWLRQ 6\VWHPV
$JUHHPHQWV &XVWRPHUDQG $SSOLFDWLRQ6FRULQJ
0DUNHWLQJ 'DWDEDVH
([WHUQDO 5HIHUHQFH $JHQFLHV
Figure 1: The current situation
The legacy infrastructure of the organisation consists of several databases, interfaces for inserting customer details and performing transactions of various types and applications for processing data and for accessing external sources of information. The heterogeneity of these legacy systems is a source of major problems including tremendous complexity, lack of sufficient documentation, co-existence of different functions serving the same purpose within the same environment, duplication of data and also confusion about which of these data are correct. Several attempts for analysing the business in the past had revealed that the existing infrastructure constrained the effectiveness of business processes. However, drastic measures could not be taken. This is a well-known problem also met by many other large enterprises handling voluminous data records and bearing strong dependencies with their information systems [Brodie and Stonebraker 1996]. Typically, the full replacement of the legacy IS would probably raise the need for parallel changes in business processes; and in addition the data conversion for supporting the new processes might be infeasible within the time limits that the business can support being without its mission-critical IS. The management of the particular organisation opted for the integration of enterprise knowledge modelling and knowledge discovery techniques. The main business objectives with respect to any business process improvement was to ensure retention of existing customers, efficiency in approaching new customers, maximum profitability of agreements and minimum contract risk through better customer understanding. The modelling approach that was used in this business application was based on the Enterprise Knowledge Development (EKD) framework. The EKD approach brings together business process modelling, and business goals within a rationale framework.
Aligning Legacy Information Systems to Business Processes
29
Recent developments of EKD are reported in [Loucopoulos, Kavakli, et al 1997; Kavakli and Loucopoulos1998]. The EKD conceptual models are organised according to four viewpoints: •
The enterprise goal sub-model expresses the concepts involved in describing enterprise objectives, enterprise issues, and their causal relationships.
•
The enterprise actor role sub-model expresses the concepts involved in modelling business processes in terms of the actors, their roles, interactions and activities in order for a business process to meet its objectives.
•
The enterprise object sub-model expresses the concepts involved in modelling physical and informational objects for the functioning of business processes and activities of individual roles.
•
The enterprise rules sub-model expresses the concepts involved in modelling the policy, and constraints, laid down by the enterprise in the way that processes, roles and resources may behave in the enterprise.
3
Enterprise Modelling for the Banking Application
It is often argued that enterprise knowledge modelling constitutes a ‘top-down’ conceptual modelling activity whereby domain experts articulate their views about existing and future business situations and requirements. This was indeed the case for the banking application, as banking personnel participated in the process of defining and validating the models for the main business functions introduced in section 2. In addition, there was a significant amount of ‘bottom up’ modelling since much of the deliberation constituted the analysis of system functionalities. The study of the bank’s support systems facilitated the understanding of business functions, regarding their purpose, and the way employees interact with each other, with customers and with the systems themselves. Much information was derived from examining existing IDEF0 documentation of system processes, interface screens and database contents. Nevertheless, it is our premise that this ‘bottom-up’ view very seldom is accurate since the actual functionality of the system is rarely that which is documented in specifications derived perhaps many years earlier. On the contrary, the knowledge discovery activity, which looks at the behaviour of the systems data, provides a safer understanding of the systems functionality (more about this in section 4). The enterprise knowledge modelling activity for the banking application considered a number of interrelated components, each corresponding to the EKD sub-models. These were: What is the purpose of each process? Who are the actors and what roles do they play within each process? What are the business objects required by the business system? What are the business rules that dictate the behaviour of the business processes?
30
4.1
Panos Kardasis and Peri Loucopoulos
Defining business goals
Regarding the first question, the business goals that guide the functioning of the company were defined as shown in Table 1. Table 1 Business Goal "Identify potential customers" "Sell products to potential customers" "Ensure maximum profitability" "Ensure low risk" "Fulfil agreement obligations"
a b c
Business Process Targeting Customers Contacting Customers Evaluating Customer Applications
d
Conducting Pay-outs
The legacy information systems (shown in Figure 1) are support systems for the corresponding business processes presented in the previous table. A set of goals for change summarised in the goal graph of Figure 2 reflects the need for improvements both in business processes and their support systems. Increase business profit Goal AND Goal Decomposition
Make more agreements
Make more profitable agreements
Decrease business running costs
Decrease processing costs
Promote the right products to the right customers
Tailor products to specific customer needs
Satisfy Customers Retain profitable customers in the long run
Improve risk management
Identify profitable customer groups Score applications effectively
Improve quality of business functions
Balance number of offers - agreements
Target customers who are more likely to respond Target customers whose applications are more likely to be accepted
Figure 2: The bank’s goals for change
'HILQLQJEXVLQHVVUROHV
Given that the overall aim of the project was to achieve customer profiling for understanding customers better, and for customising products and services to their needs and potentials, we concentrated on processes a, b and c of Table 1 for which we identified the involved actors (Table 2):
Aligning Legacy Information Systems to Business Processes
31
Table 2
Business Process
Actors
Targeting Customers Contacting Customers
Marketing manager, Database developer Customer, Sales representative, Telemarketing support system (TCS) Sales representative, Underwriter, External reference agency, TCS (Telemarketing facility for Central financial Services), AUF (Automatic Underwriting Facility)
Evaluating Customer Applications
Details about the roles that the aforementioned actors play in order to achieve the bank’s business goals were derived from the study of the IDEF0 models. The elaboration of all the available information resulted in a number of Role-Activity diagrams [Ould 1995]. The portion presented here, reference the roles of process "Underwriting Customer Applications" which are depicted in a RAD-like notation: Role: Forwarding Application for Evaluation Actor: Sales Representative
Triggering Event: Request for Underwriting Received Output: Request Information from Sales Representative Output: Convey Decision
Role: Providing Reference for Customer Actor: External Reference Agency Triggering Event: Request for Reference Received Output: Convey Reference
Figure 3: The role-activity diagrams
32
Panos Kardasis and Peri Loucopoulos
'HILQLQJEXVLQHVVREMHFWV
The bank’s business objects were mainly derived from various database tables summarised in Table 3. Table 3
Table
Contents
Customer
Personal Data, Geographic and Demographic Details, Employment Information, Household Information, Financial Status.
Product
Product Description and Value, Repayment Method, Reason of Finance.
Contact Details
Details of Phone Calls and Mailings, Details about Customer’s Response.
After identifying the principal business objects and their interrelations, we generated the business object model (Figure 4) referring to current enterprise informational entities. The latter have been associated logically, although they are derived from flat database tables, where all the attributes were interrelated through a common database key. Applicant’s Contact Details
Person
has introduced by
has partner Affinity
Prospect
Applicant
Contact Record
has
Current Contact Details
offers Persistent Personal Details
Application
Telephone Contact
Mailing Contact
Product
submits
class
aggregation
specialisation
link attribute
association
Figure 4: The main components of the business object model
'HILQLQJEXVLQHVVUXOHV
The dynamic aspect of business processes was demonstrated in terms of role interaction business rules, which are statements that control the co-ordination of activities within business roles. Activity co-ordination rules were represented as "WHEN… IF… THEN…" statements. The "WHEN…" part contains triggering events and controls, also referenced by one or more role-activity diagrams. The "IF…" part contains the preconditions for the invocation of an activity. Preconditions are logical expressions between object controls, i.e. comparison of object attributes to certain values. Finally, "THEN…" contains the activities to be initiated when the triggering event occurs and if all the preconditions are fulfilled.
Aligning Legacy Information Systems to Business Processes
33
All the identified role-interaction rules for the specific application fall into the following three categories: Non-persistent guidelines of strategic planning and marketing decisions. The term "non-persistent" implies that decisions regarding the customer groups to be contacted and the types of products to be offered are regularly revised according to market tendencies and customers’ demand. Business rules for determining which individual customers need to be contacted and when. These rules are built into the telemarketing support system. The process of contacting customers by phone is also supported by rules regarding the types of products and services that can be offered each time. Other business rules deal with the specification of agreement terms (interest rates, duration of repayment, etc.). Given that sales representatives offer services on behalf of other financial corporations, they are also bound by different rules dealing with the cross-selling of products. ’Decline’ and ’pend’ rules applicable in underwriting customer applications. When customers do not fulfil the conditions of the ’decline’ rule set, their applications are rejected immediately. When, the conditions of the ’pend’ rule set are satisfied, the application is forwarded to the underwriter for manual evaluation. The underwriter also considers a number of implicit rules, in order to coestimate the credibility of the customer, and his profitability for the bank. Examples of modelling business rules according to the EKD textual and diagrammatic notations follow: WHEN IF THEN
4
Pending_Application_Submitted (Application.Behaviour_Score > y) AND Application.Behaviour_Score < z) Request_Manual_Underwriting
Knowledge Discovery for the Banking Application
Knowledge Discovery in Data (KDD) has been defined as the non-trivial process of identifying valid, novel, potentially useful and ultimately understandable patterns in data. The KDD process usually involves data preparation, knowledge evaluation and refinement. The step of applying algorithms for extracting patterns from data is referred to as data mining. The existence of a data warehouse is a necessary "preprocessing" condition for developing knowledge discovery applications. Knowledge discovery usually starts with understanding the outcome variables, as well as the overall application universe (i.e. the data populated in the warehouse) [Tej Anand 1995]. In the case examined here, both steps were facilitated by the business object model presented in the previous section, which groups the available data attributes in an unambiguous and flexible way.
34
Panos Kardasis and Peri Loucopoulos
The mining exercise [Keane, Murray, et al 1997; Filippidou, Keane, et al 1998] was conducted upon a sample of the overall bank’s data for loans and hire purchases. As it can be seen in Figure 4, the examined data constitute history of customer applications. Apart from current contact details (phone number and address) and history of contacts (dates of calls and mailings), all the other information describes the profile of the customer the time that he submitted his application for a certain product. Applications are related to customers’: Employment Details Household Information Financial Profile Financial Status
Occupation, period with current employer, etc. Demographic information, type of occupancy, information about other family members, etc. Income, bank/building society, cards, etc. Mortgages, credit cards and loans balance, standing expenses like rent, etc.
Other critical attributes like Behaviour Score for the customer and Reason For Decline for the application are directly assigned to Application. The customer is only related to his persistent personal details, be they his/her name and surname, sex and date of birth. The outcome of data mining was a set of classification decision trees [ATTAR 1996], which represented the relation between one or more input data variables and a predefined output variable. The classification output variable was the Reason For Decline, reflecting whether an application has been accepted, rejected or just has not resulted in agreement, and the reasons why. The information represented by the generated classification decision trees is equivalent to statements of the following structure: α% of customers in (group χ for input variable κ) were accepted α% of customers in (group χ for input variable κ and group ψ for input variable λ) were rejected (reason for decline ω)
From the generated results, influences of different attributes were determined and were used to inform the enterprise knowledge models and potentially to modify the systems and the business way of working. 5
Integration Between Enterprise Knowledge Models and Discovered Knowledge
The outcome of the activities described in sections 3 and 4 is: (a)
A set of specifications for the bank’s business processes.
(b) Informal descriptions and models presenting the purpose, structure, functionality and constraints of their support IS.
Aligning Legacy Information Systems to Business Processes
(c)
35
A number of statements reflecting the conclusions from the data mining exercise (customers classification and clustering of the customer base).
We claim that none of these views is sufficient for driving the transformation of the enterprise and/or the migration of their legacy IS. However, the integration of the gained knowledge (both developed and discovered) provides an overall picture of how the bank’s business goals can be operationalised. Table 4 relates several suggestions for the improvement of the bank’s legacy IS with the business goals that these improvements may fulfil. Table 4 Goal 1: Promote the right products to the right customers a. Automated support for selecting products to be offered to individual customers according to their personal profiles b. Association of customer personal profiles with relevant data mining extracts Goal 2: Improve quality of business functions a. Accessibility of unique customer profiles by all business functions b. Automatic propagation of business tactics along all the affected business functions Goal 3: Retain profitable customers in the long-run a. Automated estimation of life-time value for individual customers b. Facility of negotiating on rejected applications coming from ’good’ customers c. Facility of re-examining rejected applications after a period of time Goal 4: Tailor products to specific customer needs a. Automated support for specifying agreement terms according to customer profiles Goal 5: Identify profitable customer groups a. Automated co-ordination of massive mailings b. Feeding of data mining extracts into mailing co-ordination systems c. Support for aligning the desired number of mailings with the acceptance probability of customer groups d. Clustering of customers according to their life-time value Goal 6: Score applications effectively a. Improvement of the algorithm for underwriting customer applications Goal 7: Decrease processing costs a. Replacement of the external customer scoring application with an internal one b. Effective management of frequency and timing for contacting individual customers Goal 8: Contact individual customers who are more likely to respond a. Feeding of data mining extracts into the telemarketing support system Goal 9: Contact individual customers whose applications are more likely to be accepted a. Feeding of customer scoring information into the telemarketing support system
Figure 5 provides a high-level view of the bank’s business processes also considering the rule sets that are used in each case. Currently, the ’decline’ and ’pend’ rule sets are the only explicitly expressed and documented sets of business rules within the bank’s functions. The ’marketing strategy’ represents approaches and tactics in organising massive mailings, being based on the analysis of the bank’s marketing database contents. The ’algorithm of the telemarketing support system’ refers to several behaviour characteristics of the ’TCS’ system. Finally, the ’scoring policy’ deals with the current method for estimating
36
Panos Kardasis and Peri Loucopoulos
customers’ credit and behaviour scores. This high-level view, depicted graphically in Figure 5 is the current way of working. To demonstrate the impact of the discovered knowledge on re-designing and improving the current way of working there were three topics of relevance: (1) Correlation between input and output variables; (2) Validation of existing business and system functions; and (3) Management of ’rule’ related knowledge. Marketing Strategy
Decline Rule Set
Pend Rule Set
PROCESS 6SHFLI\LQJ&XVWRPHU 7DUJHW*URXSV
PROCESS 8QGHUZULWLQJ &XVWRPHU$SSOLFDWLRQV
PROCESS &RQWDFWLQJ&XVWRPHUV
Scoring Policy Algorithm of the telemarketing support system
Figure 5: High-level view of the bank’s processes
Correlation between input and output variables The discovered knowledge resulted in statements of the following type: "Group x of customers is related to an application acceptance probability z%". Decision-makers are potentially aware of the existence of such rules, due to their intuition and experience. However, it is very useful to move from fuzzy estimations to more explicit and quantified correlations like the ones described above. These new rule expressions can be applied in targeting customers more accurately, by aligning the number of mailings that the company can afford to send with the usual response rate of customers. Other rule statements of the type "Group x of customers is related to an application acceptance probability z% for product y" may contribute to the maximisation of customer response by
aligning offers with customer needs. These new rules are grouped under the ’customer targeting rule set’ (Figure 6), while their integration in the new IS will satisfy goals 1, 5, 8 and 9 of Table 4. Similar clusterings may facilitate understanding of individual customers, so that maximum response to offers is ensured. A number of data mining extracts may result in business rules (’contacts co-ordination rules’) for determining which is the best time for contacting a certain type of customer, and which are the most suitable products to be offered. In this way intelligence is added to the telemarketing support system being responsible for the co-ordination of telephone contacts (goals 1 and 8 in Table 4).
Aligning Legacy Information Systems to Business Processes
37
Validation of existing business and system functions The knowledge discovery experiments showed that several operations (on business and system level) are not that much dependent on certain data attributes as previously thought. This observation raised the need for revisiting the corresponding processes and legacy support systems, and for improving them in case the problem is due to inaccurate input data, or erroneous algorithms for processing them, rather than to inaccurate data mining extracts. Such improvement efforts constitute an indicative example of cases where knowledge discovery practices are used for validating the current business and systems. Figure 6 reflects changes of the bank’s processes advocated by the previous observations: NEW! Customer Targeting Rule Set
IMPROVED!
Decline Rule Set
Pend Rule Set
IMPROVED!
PROCESS 6SHFLI\LQJ&XVWRPHU 7DUJHW*URXSV
PROCESS 8QGHUZULWLQJ &XVWRPHU$SSOLFDWLRQV
PROCESS &RQWDFWLQJ&XVWRPHUV
IMPROVED! NEW!
Scoring Policy
Contacts Co-ordination Rules
Figure 6: High-level view of the future situation
Management of rule-related knowledge Business policies, regulations, guidelines, and also discovered knowledge expressions within a large enterprise can be considered as a complex net of interactions, given that they all make reference to the same business objects and activities. Appropriate tools can facilitate the management of this knowledge, by identifying the dependencies and conflicts between rule expressions and by propagating potential changes (according to newly discovered knowledge or new business strategic decisions) along the affected business processes, the people that need to be informed and involved and the systems that need to be modified or customised accordingly (goal 2 in Table 4). 6
Conclusions
Migration of legacy IS is becoming increasingly important as organisations face significant pressures for change. Such a migration however, is a complex undertaking, requiring the commitment of large resources with uncertain results at the outset. Therefore, understanding the relationship between the needs of the business domain and the capabilities of their support IS is of critical importance. Such an
38
Panos Kardasis and Peri Loucopoulos
understanding would lead to better planning with a more evaluative way of assessing the risk of the migration task. In this paper we have put forward an approach to developing this understanding through the alignment of the business itself to its legacy IS. Central to this approach is the confluence of two techniques namely, enterprise knowledge modelling and knowledge discovery from data. The former is concerned with the development of models pertaining to business objectives, business processes and business rules whereas the latter is concerned with the development of models from the discovery of business behaviours from the existing IS. We have presented a way of utilising these techniques and demonstrated their utility in terms of a large banking application. By integrating the two sets of models it is possible to: (a) identify those parts of the legacy IS that require improvement to the extent that they will meet the stated objectives for change and (b) improve the business knowledge in terms of opportunities that may be available through the innovative exploitation of hidden knowledge. 7
Acknowledgements
The work reported in this paper has been partly supported by the commission of the European Union under the ESPRIT programme. The authors wish to acknowledge the assistance of Mr John Keane, Mrs Sofia Svinterikou and Mr Bob Scott in the insights of the data mining results. 8
References
ATTAR Software Limited. (1996) XpertRule Profiler: Knowledge from Data, 1996. Brodie, M. and Stonebraker, M. (1996) Migrating Legacy Systems, Morgan Kaufmann Publishers Inc, San Francisco California, 1996. Bubenko, J., Rolland, C., Loucopoulos, P. and de Antonellis, V. (1994) Facilitating “Fuzzy to Formal” Requirements Modelling, IEEE International Conference on Requirements Engineering, 1994. Dhar, V. and Tuzhilin, A. (1993) Abstract-Driven Pattern Discovery in Databases, IEEE Transactions on Knowledge and Data Engineering, Vol. 5, No. 6, December, 1993, pp. 926-938. Filippidou, D., Keane, J., Svinterikou, S., Murray, J., (1998) Data Mining for Business Process Improvement: Using the Hyperbank Approach, PADD98, 26-28 March 1998, London, U.K. Kavakli, V. and Loucopoulos, P. (1998) Goal Driven Business Analysis: An Application in Electricity Deregulation, CAiSE*98, 8-12 June 1998, Pisa, Italy. Keane, J., Murray, J., Scott, B. and Svinterikou, S. (1997) Preliminary Analysis of Data Mining Results, UMIST, Hyperbank WP3/T3.5/U/01, 1997. Loucopoulos, P. (1994) The F3 (From Fuzzy to Formal) View on Requirements Engineering, Ingénierie des Systèmes d'Information, Vol. 2, No. 6, 1994, pp. 639-655. Loucopoulos, P. and Katsouli, E. (1992) Modelling Business Rules in an Office Environment, ACM SIGOIS, No. August, 1992.
Aligning Legacy Information Systems to Business Processes
39
Loucopoulos, P. and Kavakli, E. (1995) Enterprise Modelling and the Teleological Approach to Requirements Engineering, International Journal of Intelligent and Cooperative Information Systems, Vol. 4, No. 1, 1995, pp. 45-79. Loucopoulos, P., Kavakli, V., Prekas, N., Rolland, C., Grosz, G. and Nurcan, S. (1997) Using the EKD Approach - The Modelling Component, UMIST, The ELEKTRA Project WP2/T2.1/UMIST/1, April 1997, 1997. Ould, M.A., (1995) Business Processes: Modelling and Analysis for Re-engineering and Improvement, John Wiley & Sons Ltd, U.K., 1995. Matheus, C.J., Chan, P.K. and Piatetsky-Shapiro, G. (1993) Systems for Knowledge Discovery in Databases, IEEE Transactions on Knowledge and Data Engineering, Vol. 5, No. 6, December, 1993, pp. 903-913. Rolland, C. and Grosz, G. (1994) A General Framework for Describing the Requirements Engineering Process, IEEE International Conference on Men, Systems, and Cybernetics, IEEE Computer Society Press, San Antonio, Texas, 1994. Tej Anand, A.T.G.I.S. (1995) Commercial Knowledge Discovery Applications, KDD95, Montreal, Canada, 1995. Yoon, J.P. and Kerschberg, L. (1993) A Framework for Knowledge Discovery and Evolution in Databases, IEEE Transactions on Knowledge and Data Engineering, Vol. 5, No. 6, December, 1993, pp. 973-979.
Automated Reverse Engineering of Legacy 4GL Information System Applications using the ITOC Workbench John V. Harrison and Wie Ming Lim Centre for Software Maintenance Department of Computer Science and Electrical Engineering The University of Queensland, Brisbane, QLD 4072, Australia E-mail: {harrison|wieming}@csee.uq.edu.au
Abstract. Most contemporary fourth-generation languages (4GLs) are tightly coupled with the relational database and other subsystems provided by the vendor. As a result, organisations wishing to change database vendors are typically forced to rewrite their applications using the new vendor’s 4GL. The anticipated cost of this redevelopment can deter an organisation from changing vendors, hence denying it the benefits that would otherwise result, for example, the exploitation of more sophisticated database technology. If tools existed that could reduce the rewriting effort, the option of changing database vendors would become more economically feasible. The ITOC project is a large collaborative research initiative between the Centre for Software Maintenance at the University of Queensland and Oracle Corporation. The primary goal of the project is to develop tools to assist in the migration of 4GL information system applications. A tool resulting from the project has been utilised to recover design information from several deployed commercial applications. This paper describes the tool, evaluates its performance when applied to these applications and provides insight into the development of “industrial strength” re-engineering tools.
1. Introduction There has been a significant investment in the development of information systems built using fourth-generation programming languages (4GLs). Although there have been considerable advances in the design of both 4GL languages and their associated development environments, most are still proprietary. There are no open industry standards, as there are with third-generation programming languages (3GLs) such as COBOL. The dependency of the customer on a particular vendor often prevents the customer from realising the benefits of newer technologies without incurring a very large redevelopment cost. Thus, in order to make application migration economically feasible, it is important to develop tools which will assist in the migration process. Most work in information system reengineering addresses either the data repository alone, or different aspects of migrating 3GL applications that access a network, hierarchical or some other legacy data repository. While there remains significant demand for tools that assist in this domain, the pace of technological advance in database systems, as well as the high level of competition amongst B. Pernici and C. Thanos (Eds.): CAiSE’98, LNCS 1413, pp. 41-57, 1998. Copyright Springer-Verlag Berlin Heidelberg 1998
42
John V. Harrison and Wie Ming Lim
database vendors, has resulted in 4GL-based information system applications now being considered as “legacy” by their organisations. The ITOC Design Recovery Tool (Harrison, et al., 1995, Berglas and Harrison, 1997, Harrison and Berglas, 1997, Harrison, et al., 1997) was developed collaboratively by the Centre for Software Maintenance at The University of Queensland, and Oracle Corporation, and has been deployed to assist with the recovery of both the application semantics and the static schema definition from Ingres ABF 4GL applications. The recovered design components are loaded into the Oracle Designer 2000 CASE repository, and can then be used to forward engineer an application in several representations, for example, HTML, Visual Basic and Oracle Developer 2000 (a “form-based” implementation environment). To the best of our knowledge, this is the first 4GL design recovery technology that has been implemented beyond a prototype stage. Existing techniques that we believed would address part of the design recovery task were determined to be insufficient when applied to deployed commercial applications. Consequently, we concur with the observations and conclusions made by (Blaha and Premerlani, 1995), which state that many techniques fail to be effective when confronted with the idiosyncrasies that occur in real applications. The next section provides a general introduction to the characteristics of 4GL information system applications using as examples the source and target development environments selected for the project. Following that, we describe the basic structure and functionality of the tool. Section four presents our results, experience, and our evaluation of the applying the ITOC tool to legacy applications. We then describe related work and conclude with a summary and future research direction.
2. Information System Application Characterics 4GL-based information systems are comprised of a user interface, application logic and a relational database management system (RDBMS). Much of the power of 4GLs is derived from the recognition that most user interfaces can be modelled as forms, which can be viewed as electronic representations of their paper-based counterparts. Forms typically contain a number of fields that correspond directly to columns in the database. The 4GL development enviroment environment is utilised to provide the (considerable) amount of user interface functionality necessary to faciliate cursor movement, e.g., to monitor cursor movement from field to field and perform basic validation. Fragments of procedural 4GL code are invoked when complex validation is necessary. For example, consider the sample Ingres ABF “order entry” form illustrated in Figure 1. In order to implement this form the programmer would declaratively specify the name, data type and position of each field, and also the fixed "boiler plate" text such as the string "Invoice Num". Most 4GLs will then automatically implement logic that permits the user to move the cursor from field to field, ensure that alphabetic characters are not entered into numeric fields, and enable multiple records such as the products purchased on the order to be scrolled in the scrolling region. Automating this functionality is an advantage of using 4GLs because implementation using a 3GL is both tedious and error prone. However, 4GL code is then required to
Automated Reverse Engineering of Legacy 4GL Information System Applications
43
perform higher level validation and processing such as ensuring that both the Billing and Shipping Customer Numbers are valid, retrieving the Customers’ Name, and calculating the “Total Delivered Cost”.
Figure 1: A Sample Ingres ABF Form The Ingres 4GL development environment requires a considerable amount of 4GL code to be written to retrieve records from the database and display them on the form, and then to update changed information. An Ingres ABF application is comprised of ABF frames. Each frame is comprised of user interface items that appear on a particular screen, termed an ABF form1, and code which implements the application logic associated with that particular screen. The code contains embedded SQL, which is typically used to retrieve values from the database into fields on the screen, and vice versa. While 4GLs reduce the amount of code required to build user interfaces, CASE tools can further improve productivity by also automating the code that is required to move data between the form and the underlying database. A developer using a CASE tool can avoid the necessity of writing code explicity to implement certain application logic and creating the user interface of each module. These components are generated automatically from a very high level description of the application, which is input into the CASE repository. For example, the Oracle CASE tool allows forms to be generated automatically based on a “Module Data Diagram”, which is made up of “Module Detail Table Usages” (MDTUs). A MDTU shows how a database table is to be used to construct a specific module. Within a MDTU, “Module Detail Column Usages” (MDCUs) show the usage of individual database columns within a particular module. Figure 2 shows a Module Data Diagram corresponding to the application illustrated in Figure 1. Each rectangle on the diagram represents a usage of a table by this module, namely a MDTU. The layout of the MDTUs in the Module Data Diagram is important as it represents the relationship between the underlying database tables. The relative placement of the MDTUs in a module ultimately determines the appearance and functionality of the generated target application. 1
Note that the term “form” is overloaded here. Ingres ABF “frames” correspond to the form construct described earlier. This is a simple example of the lack of standards in this domain.
44
John V. Harrison and Wie Ming Lim
INVOICES INVOICE_NUM ISSUE_DATE STATUS DELIVERY_NOTES REMARK CUS_CUSTOMER_NUM_SHIP CUS_CUSTOMER_NUM_BILL
Figure 2: Module Data Diagram for the Invoices Frame The CASE environment supports two types of relationships, or links, between MDTUs. These relationships are termed master-detail and look-up, which are terms that are commonly used by practitioners using different 4GL environments. Both master-detail and look-up relationships are only specified if a foreign key exists between the database tables on which the MDTUs are based. For example, a “masterdetail” relationship exists between the INVOICES and ITEMS MDTUs in Figure 2 (note the placement of the former MDTU being above the latter). This relationship implies that a foreign key exists between the ITEMS and INVOICES database tables. Similarly, a “look-up” relationship, such as the one between the CUSTOMERS and INVOICE MDTUs in Figure 2, can only be defined if a foreign key exists between the INVOICES and CUSTOMERS database tables. Look-ups are positioned to the right of the referenced MDTU on the diagram. On the basis of the above Module Data Diagram (MDD), the CASE tool can generate an application in several representations, such as the Oracle Developer 2000, HTML and Visual Basic. These representations would offer equivalent functionally to the Ingres implementation illustrated in Figure 1. The MDD graphical description is all that need be specified to enable the CASE tool to automatically generate the code required to select Invoice and Items rows from the database, look up and validate their Customer and Product Numbers, and update the database in response to user input.
3. Design Recovery From Information Systems The ITOC design recovery process involves the extraction of information from an Ingres relational database application, and the conversion of this information into Oracle CASE repository elements. An overview of the process appears in Figure 3. The ITOC tool uses information from the Ingres relational database schema as obtained from the database catalogue, the user interface (screen) definitions as extracted using the Ingres “copyapp” utility and 4GL procedural code that is contained within the source, i.e., “.osq”, files. After processing, the tool loads the
Automated Reverse Engineering of Legacy 4GL Information System Applications
45
recovered design information into the CASE repository. Executable applications, in various representations, can then be created automatically using code generators. The processing steps implemented by the ITOC tool are shown in detail in Figure 4. Each step produces objects that are defined by the ITOC analysis schema, and then used as inputs to the next step. The output of the tool are CASE elements that define the schema and modules needed to construct the Module Data Diagrams, and other design information, that corresponds to the source application. The following sections provide an overview of the processing performed at each step. Many details have been omitted due to space limitations. A more comprehensive description can be found in (Harrison and Berglas, 1997). Source 4GL: “.osq” files
Ingres 6.4 RDB Schema
Output of Copyapp Util.
ITOC Design Recovery Tool
User Preferences
Business Process Reegineering
CASE Repository
CASE Generator
Developer 2000 Application
Oracle 7 RDB Schema
Figure 3: Overview of the ITOC Design Recovery Process Data Mining Ingres Source .osq, DDL Copyapp
Parse and Link
AST Abstraction Analysis
Abstract Syntax Tree (AST)
AST Abstraction Schema
Data Flow Analysis
Query Analysis
Query Abstraction Schema
Forms Analysis
Forms Abstraction Schema
CMS Derivation
Case Meta Schema
Meta Loader
Case Repository
Analysis Meta Schema
Figure 4: Steps of the ITOC Design Recovery Process 3.1 Parse and Link The first step is to parse the source code and link uses of each identifier to its definition. This was performed using Reasoning System’s Refinery environment, including the Refine programming language and the Dialect compiler compiler. We found that, unlike many other compiler compilers, in addition to parsing the
46
John V. Harrison and Wie Ming Lim
source file given a formal grammar of the source language, Dialect automated the production of an abstract syntax tree. The Abstract Syntax Tree (AST) output consists of objects which represent instances of terminal and non-terminal nodes in the grammar, together with derived information such as the links to identifier definitions. The Language eXtension WorkBench (LXWB) (Peake, 1996) was used to extend Dialect to unify the definitions of the formal grammar and the objects that form the AST. 3.2 AST Abstraction Once the AST is created, it is compressed into a high level abstraction which is more suitable for further analysis. Defining an explicit abstraction of the source code enabled different syntactic structures to be processed in a consistent manner. For example, while most queries are represented by explicit SQL statements within the 4GL code, it was discovered that a common Ingres coding practice is to write 3GL programs which perform implicit queries by scanning plain text tables of static data on the client side of a client/server environment. AST abstraction enabled these procedure calls to be abstracted, and subsequently treated exactly the same as if they were SQL queries despite the fact that they have a radically different syntactic structure. The results of this processing phase are recorded in the AST Abstraction Schema, which is used as input into the following phase. 3.3 Data Flow Analysis The ITOC design recovery process relies on analysis of data flow in the Ingres source code. The data flow analysis is heuristic-based and differs from the conventional use of data flow analysis (Callahan 1988, Marlowe and Ryder 1990, Ryder et. al. 1990). The heuristic we adopted assumes that if a data flow exists from a source object A to a target object B, then the value of object A is coupled, and hence considered equal, to object B. Based on our experience applying the tool, which is described below, we found this heuristic to be effective. The data flow analysis phase consists of two sub-phases. Initially, all the direct data flows between variables and database columns are recorded. For example, the first line of the first SQL query variant in the code fragment below contains a data flow from I.Invoice_Num to the variable :Invoice_Num in the first query statement. Note that the data flows are recorded between each reference in the code to a database column, which we term query columns. Select From Where ... Select From Where ...
:Invoice_Num = I.Invoice_Num INVOICES I :Billing_Cust_Num = I.Billing_Cust_Num :Billing_Cust_Name = C.Cust_Name CUSTOMERS C C.Cust_Num = :Billing_Cust_Num
On the basis of the direct data flows, the transitive links that exist between two query columns, as opposed to between a query column and a variable, are computed. These transitive data flows are needed to capture the implicit enforcement of foreign
Automated Reverse Engineering of Legacy 4GL Information System Applications
47
keys, such the one between the “I.Billing_Cust_Num” and “C.Cust_Num” query columns via the “:Billing_Cust_Num” variable appearing in the code fragment. Note that this phase only computes the transitive data flows. The derivation of foreign keys is done in the following phase. 3.4 Query Analysis and Data Mining A fundamental part of the ITOC design recovery process addresses the recovery of foreign keys using information obtained from the Ingres 4GL code. This recovery is necessary as the Ingres Data Definition Language (DDL) does not support the explicit declaration of foreign keys, which form an integral part of the construction of Module Data Diagrams. Based on the data flow, both direct and indirect, recorded in the previous phase, the ITOC tool computes references between query tables, which are references to a database table appearing in a query. A reference object is created for each data flow that terminates at, or originates from, a reference to a database column that forms part of a key of a relational database table. The data flow must be from a query column, as opposed to a variable. If every part of a key is covered by a reference object, meaning there exists a data flow from some query column to each part of the given key, then a candidate foreign key is generated. The generation of the candidate foreign keys is based on heuristics about data flows. As the heuristics are not completely accurate, invalid foreign keys will be proposed. Pruning these invalid candidate foreign keys is necessary to reduces the possibility of errors in subsequent phases. The candidate foreign keys are tested using rudimentary data mining techniques applied to the original Ingres database instance. In addition to validation of candidate foreign keys, data mining is used to derive some additional information, namely: • Whether columns were deemed mandatory or optional by the source designer. Although Ingres contains null and not-null DDL definitions, their semantics are inconsistent with the target database environment. As a result the Ingres DDL cannot be used to determine this information directly. • Specific ranges for column values. For example, an Order’s status might have been restricted to “QUOTE”, “ORDERED”, or “PAID”. • Whether foreign keys participate in a one-to-one cardinality relationship. • Whether numeric fields may be contain negative values. The results of this phase are represented in the Query Analysis Schema, which would now contains objects which define the foreign keys, and other domain constraints, that were implemented in the source application. 3.5 Form Analysis Form analysis identifies which database column is used to populate each field in an Ingres frame and which fields are populated with derived values. In the CASE tool, a field can be associated with at most one database column. Ambiguities related to uniquely associating a database column, via a MDCU, to a field arise from the fact that data flows can exist between more than one query column and a given field, which we were informed often occurs in practice. For example, if a billing customer
48
John V. Harrison and Wie Ming Lim
can be the same as a shipping customer, then the billing customer’s details will be displayed in both the billing and shipping fields on our example Invoices frame. As a result, a data flow will have been created from both the shipping and billing customer names to the shipping customer field. The goal of this phase is to determine which database column should populate the shipping customer field. The solution to resolving the above problem is based on a strategy involving numerical weighting of the relative data flows. Using weighting heuristics, which are described in detail in (Tech 1996), each field can be uniquely associated with a database column. Note however, that not every field will necessarily be associated with a database column, for example, fields which get their value from a complex calculation perhaps involving several database columns. At present, the tool only highlights the existence of such fields, and provides links to their occurrence in the source code via a hypertext report viewable using a conventional web browser. The results of this phase are recorded in the Forms Abstraction subschema, and again used as input into the subsequent, and final, processing phase. 3.6 Module Analysis Having uniquely associated fields and database columns, the Module Data Diagrams can be created. Module Analysis maps each Ingres frame to a corresponding Module Data Diagram, and determines the relationships / usages between the MDTUs for the particular module. In addition to producing a mapping between each Ingres frame and a corresponding Module Data Diagram, the tool also provides the software re-engineer with alternate design options. Through analysis it was discovered that an Ingres application will often contain several distinct forms which all perform a semantically related task. For example, one Ingres form may allow the user to read Invoice details, another may allow the editing of the Invoice and a third may allow the creation of a new Invoice. In the target 4GL, the same functionality can be implemented using one form, which allows updates, queries and insertion of new records. By examining the data flows between forms, the tool is able to suggest an alternative, more succinct representation of the source application. This phase is described in (Harrison and Berglas, 1997). The remainder of this paper evaluates the effectiveness of the ITOC design recovery process described in the previous sections. The evaluation is based on the commercial and custom built applications that have been re-engineered using the ITOC tool.
4. Evaluation Of The Tool’s Performance This section describes the results of applying the ITOC tool to four Ingres ABF applications. Three are commercial applications that have been deployed. The fourth was constructed by the ITOC research group to test the tool on an application that contained constructs permissible by the language but are rarely used in practice
Automated Reverse Engineering of Legacy 4GL Information System Applications
49
according to the practitioners who participated on the project. Each application is described briefly below. The Customer Invoicing Application (CUST-INV) is the experimental Ingres ABF application, which contains various 4GL constructs that our analysis, and discussions with practitioners, indicated were obscure and only occasionally appear in a deployed commercial application. As the tool was designed to be robust in anticipation of worldwide deployment, any permissible construct had to be addressed and tested. The Curriculum Management System (CMS) is a large Ingres application (containing 120 frames) used by the Australian Technical And Further Education Institute of Australia (TAFE) to manage their curriculum. The Curriculum Monitoring Application (CMA) is a related application (10 frames) belonging to the TAFE Institute. The Contractor Monitoring System (DME) is a contractor registering and monitoring system of the Australian Department of Mines and Energy. DME contains 41 frames which record and maintain information about contractors and their licensing details. The tool’s effectiveness was measured primarily on its ability to recover a number of major design components, namely Tables, Module-Detail-Table-Usages (MDTU), Module-Detail-Column-Usages (MDCU), Foreign-Key Usages, Master-Detail Relationships (MDR), and Look-Up Relationships (LUR). Tables refer to the database tables that are referenced in a particular Ingres form. Foreign-Key Usages represent the usage of a foreign key in a given module. The remaining components were introduced in Section 2. Table 1 presents a summary of the results of the evaluation. For each application, the number of design components found in the source Ingres ABF application, by a manual line-by-line review of the code by a domain expert, appear in the columns labelled “In Source”. The quantity of each design component recovered by the ITOC tool, stored in the CASE repository and also properly displayed to the developer using the CASE environment appears in the columns labelled “Recovered”. The last column of Table 1 summarises the performance of the tool across all four applications. Although these figures give a reasonable indication of the performance of the tool, other more informative criteria may exist. This topic is currently under investigation. In addition to the analysis results based on the above criteria, we describe some additional features of the tool which facilitate the overall re-engineering process. 4.1 Evaluation Results The numerical results of the application of the ITOC tool on the four applications are shown in Table 1. Analysis indicated that omitted design information was due to one of only four reasons, namely missing foreign keys, missing MDTUs, errors in distinguishing integrity constraints, and errors displaying foreign keys that were recovered.
50
John V. Harrison and Wie Ming Lim
CUST-INV
CMA
DME
CMS
In Source
Recovered
In Source
Recovered
In Source
Recovered
In Source
Recovered
Percentage Recovered
Tables Referred
10
10
14
14
83
83
219
219
100%
Foreign Keys
10
10
6
5
30
23
205
79
46.6%
MDCUs
43
43
38
38
296
296
640
640
100%
MDTUs
15
15
14
14
78
78
295
265
92.5%
MDR
4
4
3
3
13
13
48
40
88%
LUR
6
5
3
2
17
10
154
39
31%
Table 1: Results of the Analysis of the Four Applications 4.1.1 Missing Foreign Keys As mentioned in section 3.2, we discovered that a common Ingres programming construct involved calls to 3GL procedures to retrieve code descriptions for a given code. In general, the 3GL code will contain a look-up to what is referred to as a “code table” such as the partial one illustrated in Table 2. In this example, the primary key of the code table is the columns Code_Type and Code. Code_Type
Code
Columns
SUBJ_CODE
Subj_Dtls_1
Subj_Abbr_Name
SUBJ_CODE
Subj_Dtls_2
Subj_Description
Table 2: Example Code Table In the Ingres frame, the call to one of these 3GL look-up procedures will contain a hardcoded reference to the Code_Type, for example: callproc CODE_TABLE_LOOKUP(‘SUBJ_CODE’, subj_code, BYREF(Subj_Description)); Here the calling frame passes a literal (‘SUBJ_CODE’) and a variable (subj_code) to the 3GL procedure. This results in two data flows, one to each component of the primary key of the code table. Although each component of the primary key of the code table is represented as query column appearing as an endpoint of a data flow, a reference object is not created for this data flow. Recall from section 3.3 that the source and destination of each relevant data flow must be from a query column (either directly or indirectly). As one of the data flows originates from a literal, a reference object will not be created, which results in a missing foreign key. An unrecovered foreign key subsequently results in one of the master-detail or look-up relationships being missed. With reference to the results in Table 1, this case accounts for all the missing foreign keys, and then subsequently missing look-ups, in the CMA and DME applications, and also similar missing components in the CMS application.
Automated Reverse Engineering of Legacy 4GL Information System Applications
51
4.1.2 Missing MDTUs Analysis of the CMS application revealed that all frames constructed for querying the database were of similar structure. These frames consisted of a master-detail relationship between MDTUs based on the same underlying database table. The user enters search criteria into the master frame, and the results of the query are displayed in the detail frame. The ITOC tool incorrectly corrupts the design by combining the two MDTUs into one. The error is due to incorrect formation of query table sets, which are abstractions used to represents the consolidation of similar queries and updates appearing in the 4GL code. For example, a select, update, and delete operation on the same database table, involving the same columns are represented as one query table set. This more consise representation was found to be more maintainable and would result in a better implementation after generation, as described in section 3.6. Although the query table set principle is sound in most cases, there are instances, such as the case described above, where the tool incorrectly unifies MDTUs and hence misses foreign keys and master-detail relationships appearing in the CSM application. This problem of forming correct query table sets is described in (Berglas and Harrison, 1997) and is currently under investigation. Another characteristic of the CMS application involves the population of a detail MDTU using tuples derived from a relational union of the results of several SQL queries. The version of the target CASE tool did not support the concept of relational union directly through its graphical module creation utility. As a result, in order to represent this construct in the CASE repository, the designer can either define a view definition that includes a union operation, or write a stored database (PL/SQL) procedure. At present, the ITOC tool only recovers components that are directly representable in the CASE repository, hence this construct is not recovered, which accounts for the missing MDTUs, master-detail relationships, and foreign keys that should have been recovered from the application. 4.1.3 Referential Integrity Constraints The Oracle CASE tool supports relationships between MDTUs based on foreign key constraints between database tables but does not support the more general referential integrity constraint (Elmasri and Navathe, 1994). In the DME application, there are several frames that, after design recovery, would result in MDTUs connected using referential integrity constraints, but not foreign keys constraints. An example is a look-up to a database table that references specific tuples in the target table using only part of its key. For example, consider the following ABF-variant of an SQL Select statement:
Select From Where
:Endorsee_Name = E.Endorsee_Name ENDORSEES E E.Licence_No = :Licence_No
Assuming that the primary key of the ENDORSEES table is the combination of Endorsee_Name and License_No, and that the variable :License_No contains a value from the LICENCES table. The relationship between the ENDORSEES and
52
John V. Harrison and Wie Ming Lim
LICENCES tables is based on a referential integrity constraint, but not a foreign key constraint. As a result of the restriction in the particular version of the target CASE environment used in our experimentation, a link between MDTUs based on these two database tables cannot directly enforced using the declarative mechanism provided. Consequently, the ITOC tool does not attempt to recover this information, which accounts for the missing links we expected to encounter after manual analysis of the DME application. 4.1.4 Foreign Keys Recovered But Not Displayed The ITOC tool utilises information from the user interface definitions to create MDTUs. One such component of the definition is the field sequence. The field sequence specifies the order in which the fields were created by the application developer. As a result of manual analysis, which was verified using practioners, we observed that the fields of an ABF screen that corresponds to a column of a look-up MDTU, which we term look-up fields, occur later in the sequence than fields in ABF screen that corresponds to a master MDTU, which we term master fields. Consequently, the ITOC tool employs a heuristic based on this observation that assists in distinguishing master and look-up MDTUs. However, in the case of the CUST-INV application, an anomaly occurs. A lookup field occurs before the master field. This results in the failure of the tool to create the MDTU constraint, and also accounts for the missing look-up relationship. Note however, that the associated foreign key is recovered. It is only its usage in this module that was not recovered. 4.2 Other Recovered Design Information In addition to the above, the ITOC tool recovers other design information from the source application. This recovery, and some informal observations, are described in this section. 4.2.1 User Interface Specification When the user interface definition is declaratively specified in a 4GL environment, as is most often the case, it can be almost completely recovered. Re-use of the recovered specification can be of value to an organisation in certain circumstances. If the re-engineered system retains a similar, or identical, appearance and behaviour as the source system, fostering user acceptance of the re-engineered system may be easier. In addition, the cost to retrain end-users to use the reengineered system may be reduced. The declarative interface specification includes the position of the fields on the source application’s form, the ordering sequence of the fields, the length of the fields, and the “boilerplate” text, ie., field labels, associated with each field. Text enhancement characteristics such as bold, underline, blinking, etc., are also recovered. As this information is both explicit and declaratively represented, only rudamentary information system semantic analysis is required to allow the tool to completely recover this information.
Automated Reverse Engineering of Legacy 4GL Information System Applications
53
The recovered specification is loaded into the CASE repository, and is used to create templates, which are used for generating the user interface of the target application. Even when the source user interface is text-based and the target user interface is graphical (GUI) it can be easily determined that the design of the user interface originated from the source application. The templates can also be used to create new applications, unrelated to either the source or target, that possess the same “look and feel” as the source, hence retaining consistency. 4.2.2 Business Rule Recovery Certain expressions appearing in the application logic or declarative user interface specification are both identified, and recovered, from the source application. These expressions, termed derivations and validations, can derive the value to be stored in a field and can restrict the value that will appear in a field. These expressions represent a subset of a larger set of logical expressions that complete the representation of the semantics of the application, and are termed “business rules” by practitioners. The ITOC tool analyses assignment statements such as “A := B + C”. It also analyses SQL “Select” statements such as “Select A + B, from T where C > D and (I = J or K = L)” and validation expressions such as “Minimum_Amt > Ordered_Amt”. After analysis, the tool produces a business rule report, The report contains a hyper-text (HTML) listing of each frame’s source code, displaying business rules (in italics) and references to variables that are connected to either fields or query columns (in bold), which we observed indicated the existence of a business rule. The use of the report reduces the likelihood of these rules being overlooked during the manual re-implemention phase of the re-engineering process. The practitioners who completed the development of the target application after performing design recovery using the ITOC tool found the report useful but had hoped for more extensive support for business rule recovery than provided by the tool. 4.2.3 Create, Read, Update and Delete Operations Operations for creating, reading, updating and deleting database tuples, which are collectively referred to as CRUD by practioners, are the four possible types of operations that can be defined in a CASE Module Data Diagram. This information is recovered from the source application by analysing SQL statements in the application, and is stored in the CASE repository. This allows the CASE generators to produce target applications which support the same table operation modes as the source application. This information is both explicit and declaratively represented in the 4GL code, hence only rudamentary information system semantic analysis is required for full recovery. 4.2.4 Metric Report The ITOC tool contains a utility for estimating the effort required to complete reengineering of an application after design recovery using the ITOC tool is complete. The report is based on heuristics provided by the practioners that reason with the
54
John V. Harrison and Wie Ming Lim
number of activations appearing in a frame. Activations, which are also referred to as triggers, are event-activated procedural code segments appearing in a 4GL application. The heuristics also reason with the number of activations associated with each field and the number of occurances of a program variable in non-trivial expression. The practioners indicated that the report proved useful. A more detailed description of this report appears in (Tech 1996).
5. Related Work This section provides a summary of research that relates closely to the work described here. It is loosely classified into that which only attempts to analyse data, and that which also attempts to migrate application code. (Petit et al., 1994) proposed a design recovery method to extract an entityrelationship (EER) schema from a relational database extension and SQL queries. Like ITOC, the method does not require that the same attribute in different relations have the same name nor that the relations are in 3NF. (Andresson, 1994) proposed a similar approach. However, like much of the related work, no indication is provided as to whether these methods were actually implemented. (Chiang et al., 1994) proposed a method to extract an EER model from the database extension. It assumes that the database is in 3NF, that there is consistent naming of attributes, that there are no erroneous data values and that inclusion dependencies are available. A prototype implementation is described but no results of utilising the prototype are provided. (Signore et al., 1994) describe an expert system prototype for rebuilding a conceptual schema from a logical schema. Unlike ITOC, the approach does not utilise the database extension, so it does not verify the results on the inferencing process. (Rugaber and Doddapaneni, 1993) attempted to automate the extraction of and SQL schema from a COBOL program that used flat ISAM files. Like ITOC, the recovered database design information was loaded into a CASE tool prior to forward engineering into SQL. Unlike ITOC, however, the structure of the application programs are not recovered. Other researchers, such as (Navathe and Awong, 1988), (Markowitz and Makowski, 1990), (Winans and Davis, 1990), (Fong and Ho, 1993), (Premerlani and Blaha, 1994), (Campbell and Halpin, 1996), and (Lim and Harrison, 1996), have described similar approaches for extracting conceptual models from either relational, hierarchical or network data models. The next class of related work we review attempts to migrate the application programs as well as just the static schema. (Zoufaly et al, 1995) describe an emulation approach for migrating RPG to Oracle PL/SQL and Forms 3. Low-level transformation, implemented using their own highlevel language, Logistica are used to directly translate RPG statements to Forms 3. Although an implementation is reported, the target system’s representation is at the same low level of abstraction as the RPG source which does not facilitate understanding and maintenance. COBOL/SRE (Engberts, et al., 1993) is described as a “renovation and reengineering” environment. The environment produces reports and support
Automated Reverse Engineering of Legacy 4GL Information System Applications
55
sophisticated browsing and enhancement to an existing COBOL system. Unlike ITOC, this tool only supports forward engineering back into COBOL and so does not raise the level of abstraction. No results were presented as to the effectiveness of the environment. (Karakostas, 1992) describes a similar system. (Vermeer and Apers, 1995) described an approach for recovering design information from 3GL application programs based on the observation that “C” data structures that store the result of SQL queries can serve as object schemata. No indication is given that a prototype has been constructed to evaluate the feasibility of the approach. (Hainaut et al., 1995), proposed a design recovery methodology and generic architecture for performing the extraction of an enhanced ER schema from a database followed by the conceptualisation of the resulting data structures. The ITOC tool described in this paper is novel in that it recovers the structure of 4GL applications at a high level so that new application programs can be generated using a CASE tool. It is also unusual because it has been applied to commercial legacy information systems and the results evaluated by experienced information system developers.
6. Conclusion In this paper, we have outlined the functionality of the ITOC tool and described the results obtained from applying the tool to commercial deployed legacy information systems. The experiment conducted indicates that the approach can be successfully used to recover certain fundamental design components, such as module data diagrams that capture the semantics of the information system application, and also database constraints that are not explicit in the data definition language. The experimentation revealed that “industrial-strength” tools can be constructed using the approach as both a contemporary commercial CASE repository product and (deployed) commercial information systems implemented using a contemporary, commercial 4GL were utilised. As we expected, there were some design constructs that the tool should have recovered from the source application but failed to do so. However, it is likely that these deficiences can be corrected and do not represent a fundamental flaw in the approach. Consequently, we encourage other researchers to utilise it. Our experience indicates that it is essential to apply software re-engineering tools on deployed, commercial applications to demonstrate their true effectiveness. It is also beneficial to have the output of the tools reviewed by experienced practitioners to gain useful feedback for tool improvement. From this feedback, we learned that the ITOC approach must be improved in the area of business rule recovery. Ongoing research involves the monitoring of practitioners who are utilising the ITOC tool to determine what proportion of the total re-engineering effort is eliminated due to its use. We are also investigating metrics that can be used to estimate this effort, which will be extracted automatically from a source application via a utility that includes the source 4GL language model. Finally, we also plan to study the end-user’s acceptance of the re-engineered systems produced using the ITOC tool and the costs of retraining these users.
56
John V. Harrison and Wie Ming Lim
Acknowledgements: The authors wish to recognize the contributions of Professor Paul Bailes, Dr. Anthony Berglas and Mr. Ian Peake ( CSM), and also Mr. Blair Layton and Mr. Ronald Bradford (Oracle Corporation).
References Andersson, M., Extracting an Entity Relationship Schema from a Relational Database through Reverse Engineering. Proc. of the 13th Entity-Relationship Conference, Manchester, UK, Dec. 1994, pp. 403-419. Berglas A. and Harrison, J.V., Evaluation of the ITOC Information System Design Recovery Tool, Proc. of Fifth International Workshop on Program Comprehension, Michigan, March 1997, pp 176-182. Blaha, M.R. and Premerlani, W. J., Observed Idiosyncracies of Relational Database Designs, Proc. of Second Working Conference on Reverse Engineering, Toronto, Ontario, July 1995, pp. 116-125. Callahan, D., The Program Summary Graph and Flow-sensitive Interprocedural Data Flow Analysis, Proc. of the SIGPLAN’88, Conference on Programming Language Design and Implementation, Atlanta, Georgia, June 1998, pp. 22-24. Campbell, L.J. and Halpin, T.A., The Reverse Engineering of Relational Databases. Proceedings of the 5th Workshop on the Next Generation of CASE Tools, Utrecht, June 1994, pp 50-66. Chiang, H.L., Barron, Terrence M., and Storey, V.C., Reverse engineering of relational databases: Extraction of an EER model from a relational database. Data & Knowledge Engineering 12 (1994), pp. 107-142. Elmasri R. and Navathe S.B., Fundamentals of Database Systems. 2nd Ed. The Benjamin/Cummings Publishing Comp, Inc. 1994. Engberts, Andre, Kozaczynski, Liongosari, Edy and Ning, J.Q., COBOL/SRE: A COBOL System Renovation Environment, Proc. of the 6th Intl. Workshop for Computer-Aided Software Engineering, Ed. H. Lee and Thonas Reid, Singapore, July 1993, pp. 199-210. Fong, J. and Ho, M., Knowledge-based approach for abstracting hierarchical and network schema semantics. Proc. of the 12th Entity Relational Approach, Arlington, Texas, Dec. 1993, pp. 508-519. Harrison, J.V., Bailes P.A., Berglas A., Peak I., Re-engineering 4GL-based Information System Applications. Proc. of the Asia Pacific Software Engineering Conference, Brisbane, Dec. 1995, pp. 448-457. Harrison, J.V. and Berglas A., Data Flow Analysis with the ITOC Information System Design Recovery Tool, Proc. of Automated Software Engineering Conference, Incline Village, Nevada, USA, Nov. 1997, pp. . Harrison, J.V., Bailes P.A., Berglas A., Peak I., Legacy 4GL Application Migration Via Knowledge-Based Software Engineering Technology: A Case Study , Proc. of Australian Software Engineering Conference, Sydney, Oct. 1997, pp. 70-78. Hainaut, J.L., Englebert, V., Henrard, J., Hick, J.M., Roland, D., Requirements for Information System Reverse Engineering Support, Proc. of the 2nd Working Conference on Reverse Engineering, Toronto, Ontario, Canada, July 1995, pp 136-145.
Automated Reverse Engineering of Legacy 4GL Information System Applications
57
Lim, W.M. and Harrison, J.V., An Integrated Database Re-engineering Architecture A Generic Approach, Proc. of Australian Software Engineering Conference, Melbourne, Aug. 1996, pp. 146-154. Karakostas, V., Intelligent Search and Acquisition of Business Knowledge from Program. Software Maintenance: Research and Practice, Vol. 4, (1992), pp. 1-17. Markowitz, V.M. and Makowsky, J. A., Identifying Extended Entity-Relationship Object Structures in Relational Schemas. IEEE Transactions on Software Engineering. Vol. 16, No. 8. Aug. 1990, pp. 777-790. Marlowe, T.J., and Ryder, B.G., 1990. An efficient hybrid algorithm for incremental data flow analysis. In Conf. Recored of the 17th Annual ACM Symp. On Principles of Programming Languages (San Francisco, Jan.), ACM Press, pp. 184-196. Navathe, S.B. and Awong, A.M., Abstracting Relational And Hierarchical Data With A Semantic Data Model. Proceedings of 8th Entity Relational Approach: A bridge to the future, New York, Nov. 1988, pp.305-333. Peak, I.., User’s Guide to the Language Extension Work Bench Version 1, Technical Report 387, Centre for Software Maintenance, Department of Computer Science, University of Queensland, Australia, March 1996. Petit, J.M., Toumani, F., Boulicaut, J.F., Kouloumdjian, J., Using Queries to Improve Database Reverse Engineering. Proc. of the 13th Intl. Conf. on Entity Relational Approach, Manchester, Springer-Verlag, 1994, pp. 369-386. Premerlani, W.J. and Blaha, M.R., An Approach for Reverse Engineering of Relational Databases, Communication of the ACM, Vol.37, No.5 May 1994, pp. 42-49. Rugaber, S. and Doddapaneni, S., The Transition of Application Programs from COBOL to a Fourth Generation Language, In Proc. of Intl. Conference on Software Maintenance, 1993, pp. 61-70. Ryder, B.G., Landi, W., and Pande, H. Profiling an incremental data flow analysis algorithm. IEEE Trans. Software Eng., 16, 2: 1990, pp. 129-140. Signore, Oreste, Loffredo, Mario, Gregori, M., and Cima Marco, Reconstruction of ER Schema from Database Applications: a Cognitive Approach, Proc. of the 13th Intl. Conf. on Entity Relational Approach, Manchaster, Springer-Verlag, 1994, pp. 387-402. Technical Report on the Ingres to Oracle Design Recovery Workbench, Department of Computer Science and Electrical Engineering, University of Queensland, 1996. Vermeer, M.W.W. and Apers, P.M.G., Reverse engineering of relational database applications. Proc. of the 14th International Conference on Object-Oriented Entity Relationship Modelling, Gold Coast, Australia, Dec. 1995, pp. 89-100. Winans, J., and Davis K.H., Software Reverse Engineering from a Currently Existing IMS Database to an Entity-Relationship Model, Proceedings of 9th Entity Relationship Approach, Lausanne, Switzerland, Oct. 1990, pp. 333-348. Zoufaly, Federico, Araya, Carlos, Sanabria I. and Bendeck, F., RESCUE: Legacy System Translator, In Proc. of the Second Working Conference on Reverse Engineering, Toronto, Ontario, July 1995, pp 39-50.
Adapting Function Points to Object Oriented Information Systems? G. Antoniol1 , F. Calzolari1 , L. Cristoforetti1 , R. Fiutem1 and G. Caldiera2 1
I.T.C.-I.R.S.T., Via alla Cascata, I-38050 Povo (Trento), Italy tel. +39 461 314-444 e-mail: antoniol,calzolar,cristofo,[email protected] 2 University of Maryland, Dept. of Computer Science, College Park, Maryland 20742, USA tel. +1 301 405-2707 e-mail: [email protected]
Abstract. The object oriented paradigm has become widely used to develop large information systems. This paper presents a method for estimating the size and effort of developing object oriented software. The approach is analogous to function points, and it is based on counting rules that pick up the elements in a static object model and combine them in order to produce a composite measure. Rules are proposed for counting “Object Oriented Function Points” from an object model, and several questions are identified for empirical research. A key aspect of this method is its flexibility. An organization can experiment with different counting policies, to find the most accurate predictors of size, effort, etc. in its environment. “Object Oriented Function Points” counting has been implemented in a Java tool, and results on size estimation obtained from a pilot project with an industrial partner are encouraging.
Keywords: Object oriented design metrics, function points, size estimation.
1
Introduction
Cost and effort estimation is an important aspect of the management of software development projects and it could be a critical point for complex information systems. Experience shows how difficult is to provide an accurate estimation: in literature [18] an average error of 100% is considered to be “good” and an average error of 32% to be “outstanding”. Most research on estimating size and effort has dealt with traditional applications and traditional software development practices, while few works have been experimented for object oriented (OO) software development. ?
This research was funded by SODALIA Spa, Trento, Italy under Contract n. 346 between SODALIA and Istituto Trentino di Cultura, Trento, Italy.
B. Pernici and C. Thanos (Eds.): CAiSE’98, LNCS 1413, pp. 59–74, 1998. c Springer-Verlag Berlin Heidelberg 1998
60
G. Antoniol, F. Calzolari, L. Cristoforetti, R. Fiutem and G. Caldiera
This paper presents a method for estimating the size and development effort of object oriented software, supported by a tool, implemented in Java. The proposed approach, that we call “Object Oriented Function Points” (OOFP), is based on an adaptation for object oriented paradigm of the classical Function Point (FP) methodology [2]. As shown in Figure 1, we will measure Object Oriented Function Points, and correlate them with actual system size and development effort to identify estimation models tailored for a specific environment. One of the advantages of this approach is that different estimation models can be developed for different stages of a software project, as soon as the software artifact becomes more detailed while the project goes on. The OOFP Counter, the Java tool that implements the proposed approach, provides a way to finely tune the counting rules by setting several parameters related to which counting policy is better suited for a given software project. This paper is organized as follows: Section 2 explains how we map main concepts of function points to object oriented software. The rules for counting Object Oriented Function Points are then described in Section 3, with emphasis on different counting policies that can be adopted. Section 4 presents the OOFP Counter, the tool developed to automatize the counting process. This tool has been used to produce results for an industrial pilot project, focused on size estimation, reported in Section 5. Finally, conclusions are drawn.
2
Object Oriented Function Points
Since they have been proposed in 1979 [1], function points (FP) have become a well known and widely used software metric. Despite some concerns [10,11,12,17], practitioners have found FPs to be useful in the data processing domain, for which they were invented. Function points are available at the specification phase since they are based on the user’s view of software functionality. FPs are generally considered to be independent from the technology used to implement the solution. The key features of function points are that they are available early, and they are a measure of the problem independent from any particular implementation. The International Function Point Users Group (IFPUG) publishes guidelines to standardize their definition [6]. Several variants have been proposed to extend FPs use to other domains (see [5] for a survey). Since OO paradigm had become widely adopted to design large information systems, different attempts have been proposed to adapt function points concepts to object oriented software, in order to exploit the understanding gained with function points in their traditional domain. In the object oriented approach, an object model uses classes and their interrelationships to represent the structure of a system. While the development proceeds the object model evolves: in addition to the problem-related classes, the model includes design- and implementation-oriented classes with new inheritance relationships. These changes do not concern the user, but reflects the developer’s
Adapting Function Points to Object Oriented Information Systems System Requirements Definition
#FP USER
OO Analysis
OO Design
Implementation
#OOFP
#OOFP
Code
DESIGNER
61
PROGRAMMER
Fig. 1. Measures in the software development process.
view of the system. A measure derived from the object model should be now a better predictor of development size and effort. The OOFP approach enables a smooth transition from the user’s view to the developer’s view, and the same methodology can be used to measure the object model at each stage, as shown in Figure 1.
2.1
Mapping function points to object oriented software
Object model, dynamic model, and functional model may be used to represent information about object oriented software [14]. The object model is usually the first to be developed, and it is the only one that describes the system using specifically object-oriented concepts. We focus our attention to object model to map traditional FP concepts to OOFP, translating logical files and transactions to classes and methods. A Logical File (LF) in the function point approach is a collection of related user identifiable data. Since a class encapsulates a collection of data items, it seems to be the natural candidate for mapping logical files into the OO paradigm. Objects that are instances of a class in the OO world correspond to records of a logical file in data processing applications. In the FP method the application boundary identifies Internal Logical Files (ILFs) (logical files maintained by the application) and External Interface Files (EIFs) (referenced by the application but maintained by other applications). In the OO counterpart, we could consider external classes encapsulating non-system components, such as other applications, external services, and library functions. Classes within the application boundary correspond to ILFs. Classes outside the application boundary correspond to EIFs. In the OO paradigm operations are performed by methods (which are usually at a more fine-grained level than transactions). Since object models rarely contain the information needed to tell whether a method performs an input or an output or is dealing with an enquiry, we simply treat them as generic Service Requests (SRs), issued by objects to other objects to delegate some operations. Issues such as inheritance and polymorphism affect the structure of the object model, and how the model should be counted. This problem will be addressed in Section 3.1.
62
2.2
G. Antoniol, F. Calzolari, L. Cristoforetti, R. Fiutem and G. Caldiera
Related work
Several authors have proposed methods for adapting function points to object oriented software. In [15] classes are treated as files, and services delivered by objects to clients as transactions, while in [19] each class is considered as an internal file, and messages sent across the system boundary are treated as transactions. Sneed [16] proposed object points as a measure of size for OO software. Object points are derived from the class structures, the messages and the processes or use cases, weighted by complexity adjustment factors. A draft proposal by IFPUG [7] treats classes as files, and methods as transactions. Fetcke [3] defines rules for mapping a “use case” model [9] to concepts from the IFPUG Counting Practices manual, but no attempt has been made to relate the results to other metrics, such as traditional function points, lines of code, or effort. The key aspect of our approach is its flexibility. For example, Fetcke [3] defines that aggregation and inheritance should be handled in a particular way. We define several options (one of which is Fetcke’s approach) and leave it to the user to experiment which parameter settings produce the most accurate predictors of size, effort, etc. in its environment. Thus we have a method which can be tailored to different organizations or environments. Moreover, the measurement is not affected by subjective ratings of complexity factors, like those introduced in classical function point analysis. Finally, the OOFP Counter will automatically count OOFPs, for a given setting of parameters.
3
Measurement Process
OOFPs are assumed to be a function of objects comprised in a given object model D (D can be that produced at design stage or extracted from the source code) and they can be calculated as: OOF P = OOF PILF + OOF PEIF + OOF PSR where: OOF PILF =
X
WILF (DETo , RETo )
o∈A
OOF PEIF =
X
WELF (DETo , RETo )
o∈A /
OOF PSR =
X o∈A
.
WSR (DETo , F T Ro )
Adapting Function Points to Object Oriented Information Systems
63
A denotes the set of objects belonging to the application considered and o is a generic object in D. Dets, Rets and Ftrs are elementary measures to be calculated on LFs and SRs and used to determine their complexity through the complexity matrixes W . Such meaasures are further detailed in Sections 3.2 and 3.3.
Det
SC
L,A,H
ILF()
7,10,15 OOFP
Ret Identification Module
OO Design
ILF
GB Det AB
EIF()
L,A,H
Ret
MB
Det SR()
5,7,10 OOFP
L,A,H
3,4,6
OOFP
Σ EIF
OOFP
SR
Ftr
Fig. 2. OOFP computation process.
Counting OOFPs is a four steps process: 1. The object model is analyzed to identify the units that are to be counted as logical files. 2. The complexity of each logical file and service request is determined. Structural items are mapped to complexity levels of low, average, or high. 3. The complexity scores are translated into values. 4. The values are summed to produce the final OOFP result. Figure 2 outlines the counting process. The counting rules used in these steps are described in Sections 3.1 to 3.3, while Section 4.1 explores the effect of counting classes in different ways. 3.1
Identifying logical files
Classes are generally mapped into logical files. However, relationships between classes (aggregations and generalization/specializations in particular) can sometimes require to count a group of classes as a single logical file. Different choices of how to deal with aggregations and generalization/specialization relationships lead to different ways to identify logical files. In what follows we are going to present the four different choices we identified: a simple example taken from [4] will support explanation. 1. Single Class: count each separate class as a logical file, regardless of its aggregation and inheritance relationships (Figure 3). 2. Aggregations: count an entire aggregation structure as a single logical file, recursively joining lower level aggregations (Figure 4).
64
G. Antoniol, F. Calzolari, L. Cristoforetti, R. Fiutem and G. Caldiera
3. Generalization/Specialization: given an inheritance hierarchy, consider as a different logical file the collection of classes comprised in the entire path from the root superclass to each leaf subclass, i.e. inheritance hierarchies are merged down to the leaves of the hierarchy (Figure 5). 4. Mixed: combination of option 2 and 3 (Figure 6).
MapSite Enter
Maze AddRoom RoomNo
Room
Wall
Door
Enter
isOpen Enter
roomNumber Enter SetSide GetSide
Fig. 3. Single class ILFs.
MapSite Enter
Maze AddRoom RoomNo
Room
Wall
Door
Enter
isOpen Enter
roomNumber Enter SetSide GetSide
Fig. 4. Aggregations ILFs. Merging superclasses into subclasses makes intuitive sense. It seems right to count leaf classes, with their full inherited structure, since this is how they are instantiated. Dividing a user-identifiable class into an aggregation of sub-classes is an implementation choice. Thus from the point of view of the function point measurement philosophy, the OOFP value should not be affected. From this perspective, the aggregation structure should be merged into a single class and counted as a single logical file. Merging aggregations or not seems to depend on whether the user’s or designer’s perspective is chosen. However, a hybrid solution can be adopted as well, flagging on the design which aggregations must be considered as a unique entity and thus must be merged.
Adapting Function Points to Object Oriented Information Systems
65
MapSite Enter
Maze AddRoom RoomNo
Room
Wall
Door
Enter
isOpen Enter
roomNumber Enter SetSide GetSide
Fig. 5. Generalization/Specialization ILFs. MapSite Enter
Maze AddRoom RoomNo
Room
Wall
Door
Enter
isOpen Enter
roomNumber Enter SetSide GetSide
Fig. 6. Mixed ILFs.
3.2
Complexity of Logical Files
For each logical file it is necessary to compute the number of DETs (Data Element Types) and RETs (Record Element Types). Counting rules depend on whether it is a simple logical file, corresponding to a single class, or a composite logical file, corresponding to a set of classes. For simple logical files: – One RET is counted for the logical file as a whole, because it represents a “user recognizable group of logically related data” [6]. – Simple attributes, such as integers and strings, are considered as DETs, as they are a “unique user recognizable, non-recursive field of the ILF or EIF” [6]. – Complex attributes are counted as RETs. A complex attribute is one whose type is a class (i.e. “a user recognizable subgroup of data elements within an ILF or EIF” [6]) or a reference to another class. – A single-valued association is considered as a DET (IFPUG suggests counting a DET for each piece of data that exists because the user requires a relationship with another ILF or EIF to be maintained[6]). – A multiple-valued association is considered as a RET, because an entire group of references to objects is maintained in one attribute. – Aggregations are treated simply as associations.
66
G. Antoniol, F. Calzolari, L. Cristoforetti, R. Fiutem and G. Caldiera
For composite logical files: – Using the rules for simple logical files, except for the handling of aggregations, DETs and RETs are counted separately for each class within the composite. – In a composite logical file aggregations represent a subgroup. One RET, assigned to the container class, is counted for each aggregation, whatever its cardinality. One more RET is also counted for the logical file as a whole. – The individual DETs and RETs are summed to give an overall total for the composite logical file. When the DETs and RETs of a logical file have been counted, tables (derived from those given in the IFPUG Counting Practices Manual Release 4.0 [6] for ILFs and EIFs) are used to classify it as having low, average, or high complexity. 3.3
Complexity of Service Requests
Each method in each class is considered: abstract methods are not counted, while concrete methods are only counted once (in the class in which they are declared), even if they are inherited by several subclasses. If a method is to be counted, the data types referenced in it are classified as simple items (analogous to DETs in traditional function points) for simple data items referenced as arguments of the method, and complex items (analogous to File Types Referenced (FTRs) in traditional function points) for complex arguments [2]. Again tables are used to classify the method as having low, average, or high complexity. Notice that sometimes the signature of the method provides the only information on DETs and FTRs. In such a case, the method is assumed to have average complexity. 3.4
An example
The counting procedure for each individual class gives the DETs and RETs shown in Figure 7, while Table 1 shows ILF and SR contribution to OOFP counting. Since service requests (methods) are only counted once, it does not matter how the classes are aggregated into logical files. Because the signatures are unknown for the methods in the example, each method is assumed to have average complexity. Values in third and fifth columns show the results of applying IFPUG 4.0 complexity tables with each variant. The value 7 is rated as Low and it is weighted 4. For more details about how counting rules have been applied the interested reader could refer to [2]. The highest OOFP count comes when each class is counted as a single ILF. All the other variants have the effect of reducing the OOFP value, as they reduce the number of ILFs. Although there is an increase in DETs/RETs in the merged ILFs, it is not enough to raise the ILF complexity to higher values. For this example, and for the pilot project that will be presented in Section 5, the complexity of each ILF and SR are always determined to be low. The tables
Adapting Function Points to Object Oriented Information Systems
67
MapSite
DET=1 RET=1
Enter
Maze AddRoom RoomNo
DET=0 RET=2
Room
Wall
Door
Enter SetSide GetSide
Enter
isOpen Enter
DET=2 RET=2
DET=0 RET=1
DET=1 RET=1
roomNumber
Fig. 7. DET/RET computation for LFs on the example system. Single Class Aggregation Generalization/Specialization Mixed
ILF ILF OOFP 5 35 4 28 4 28 3 21
SR SR OOFP Total OOFP 7 28 63 7 28 56 7 28 56 7 28 49
Table 1. ILF and SR complexity contribution.
used to determine complexity are based on those from the IFPUG Counting Practices Manual [6], in which quite large numbers of RETs and DETs are needed to reach average or high complexity (for example, to obtain an average complexity weight an ILF needs a DET value between 20 and 50 and a RET value between 2 and 5). On the data available to us so far, we suspect that recalibration of the OOFP tables for logical files might improve the accuracy of OOFP as a predictor of size, but further experimentation is needed on this topic.
4
The OOFP Counter Tool
We have developed the OOFP Counter tool, presented in Figure 8, to automate the OOFP counting process. This tool has been implemented using Java. The OOFP Counter inputs Abstract Object Language (AOL) specification of the object oriented model. AOL is a general-purpose design description language capable of expressing concepts of OO design. It has been adopted in order to keep the tool independent of the specific CASE tool used. AOL is based on the Unified Modeling Language [13], which represents de facto a standard in object oriented design. The OOFP Counter tool parses AOL specification and produces an abstract syntax tree representing the object model. The parser also resolves references to identifiers, and performs some simple consistency checking (e.g. names referenced in associations have been defined).
68
G. Antoniol, F. Calzolari, L. Cristoforetti, R. Fiutem and G. Caldiera
CASE Tool
Tool Output
OOFP Counting Rules
AOL Specification
AOL Translator
AST OOFP Counter
AOL Parser OOFP_Counter
ILF
SR
OOFP
Fig. 8. The OOFP Counter tool.
To improve portability, the AOL parser and the OOFP counter, the two parts of the OOFP Counter tool have been implemented in Java. For the project presented in Section 5, OMT/STP [8] has been used as CASE tool; an automatic translator to convert from OMT/STP output to AOL specifications has been implemented. 4.1
Parameters setting
The OOFP Counter works on the abstract syntax tree and implements the OOFP Counting Rules described in section 3. It is possible to set several parameters, that may influence the counting policy: – – – – –
ILF counting strategy (see Section 3.1) External classes inclusion Private methods counting; Private attributes counting; Values of DET, RET, and FTR thresholds between low, average, and high complexity.
Parameter setting might be guided by some philosophy. For example, from a traditional function point perspective one would wish to count only user-visible abstractions, ignoring all implementation aspects. This might mean selecting the Mixed strategy for grouping classes into logical files, counting only those methods which are publicly visible and related to classes at the system boundary, and giving full weight to classes whether they are reused or not. ¿From a designer’s point of view, one might want to take account of all implementation details, in an attempt to get an accurate estimate of development effort. This might mean counting each class as a separate logical file, including all methods and attributes, and reducing the weight given to reused classes. Different parameter settings could be tried on a purely experimental basis in order to identify that company specific profile that gives the best overall performance for estimating size or effort.
Adapting Function Points to Object Oriented Information Systems
5
69
An industrial case study
The described methodology has been applied in an industrial environment. Our first study is of the relationship between the OOFP measure of a system and its final size in lines of code (LOC), measured as the number of non-blank lines, including comments. Size estimation is important, since it is needed for most effort estimation models, thus we can make use of existing models that relate size to effort. Eight completed (sub-)systems were measured, for which both an OO design model and the final code were available. All were developed in the same environment, using the C++ language. Table 2 shows the size of each system, spreading from about 5,000 to 50,000 lines of code. Table 2 also shows the OOFP count for each system, using each of the four different strategies for identifying logical files.
System LOC Single Class Aggregation Generalization Mixed (SC) (AB) (GB) (MB) A 5089 63 63 35 35 B 6121 476 469 462 455 C 15031 284 284 270 270 D 16182 1071 1057 1057 1043 E 21335 562 513 548 499 F 31011 518 403 483 368 G 42044 1142 1100 1124 1072 H 52505 2093 1947 1872 1737
Table 2. System sizes and OOFPs.
The four OOFP series are strongly correlated each other, with all correlations within the .992 − .998 range (Pearson), the lowest corresponding to SC vs MB. As shown in Table 2, differences between the methods become appreciable only for the projects with large LOC values. Several regression techniques were considered to model the LOC-OOFP association. Given the reduced size of the database, a leave-one-out cross-validation procedure was used to achieve unbiased estimates of predictive accuracy for the different models. Model error was expressed in terms of normalized mean squared error (NMSE): each model was trained on n − 1 points of the data base L (sample size is currently n = 8) and tested on the withheld datum; NMSE is obtained over L normalizing over the sample variance of the observed values (µy = mean(y)). The small size of the database and a limited knowledge of LOC measures validity required the use of simple models capable to handle non obvious outliers in the response variable LOC. In this study, the basic least squares linear fit was compared with resistant techniques. Regression estimates based on least square
70
G. Antoniol, F. Calzolari, L. Cristoforetti, R. Fiutem and G. Caldiera
Table 3. Model performance for linear regressors (lms and l1s) and robustified methods (rregs and rlms). The normalized mean squared error (NMSE) and the normalized mean absolute error (NMAE) are estimated by cross-validation.
Adapting Function Points to Object Oriented Information Systems
71
minimization are in fact sensitive to outliers in the response variable when the error distribution is not Gaussian. Robust regression techniques may improve the least-squares fit and handle model inadequacies due to unusual observations. First linear models (lms) based on the minimization of the sum of squares of the residuals were developed for each ILF selection method. Least absolute deviation, based on L1 error was also applied (l1s) . The regressor is build minimizing the sum of the absolute values of the residuals to resist the effect of large error values. A family of M-estimators was therefore considered (rregs and rlms). The basic idea of M-smoothers is to control the influence of outliers by the use of a non-quadratic local loss function which gives less weight to “extreme” observations. Non-linear modelling was also attempted, expecting instability and lack of convergence due to the sample size.
NMSE Comments 0.368 – 0.367 – 0.367 – 0.480 converged after 50 steps) 0.381 – 0.378 – 0.357 c = 1.25 0.337 c = 0.80 0.380 – 0.380 –
Table 4. Model performances for different weighting functions of the Mestimator rreg. Results are given for the GB selection method only.
Estimated model accuracy for each model yˆ = b0 + b1 x of each experimented family is collected in Table 3, parametrized over ILF selection methods and type of regressor. The model coefficients b0 and b1 are indicated as computed from the full data set. Estimated R-squared measure is also included for the linear models for comparison with other results separately obtained on these data. A point of concern is the inclusion of an intercept term b0 in model: it is reasonable to suppose the existence of support code unreferred to the functionalities being counted, and prediction is improved whith the term. However, the intercept term is not significant in a non-predictive fit of the data. More important, the fact that the intercept term is always larger than the first LOC value might indicate poor fit for small OOFP values. It would be interesting to apply a Bayesian procedure to select the intercept from given priors. The estimates for different weighting functions of the M-estimator are listed in Table 4.
G. Antoniol, F. Calzolari, L. Cristoforetti, R. Fiutem and G. Caldiera
50000
72
30000 10000
20000
LOC
40000
LOC = 7183.4 + 25.6 GB
rreg−logistic−0.8 lm 0
500
1000
1500
GB
Fig. 9. The rreg-logistic-GB model (c=0.8) compared with the linear model lmGB.
The best predictive accuracy (NMSE= 0.337) was achieved by the rreglogistic-GB model with tuning parameter u = .8, corresponding to the linear predictor LOC = 7183.4 + 25.6 GB. As shown in Figure 9, the rreg-logistic-GB model is very close to the basic linear model lm-GB, whose equation is LOC = 7435.1 + 25.2 GB. As the GB method is consistently better for all models and for both the predictive error measures NMSE and NMAE, these results indicate that the choice of ILF selection method may influence prediction. Lowess, supersmoother and predictive splines have been also tested and showed instability of convergence due to the small sample size. Although more experimental work is needed, obtained results are encouraging for size estimation.
6
Conclusions
This paper shows how the concepts of function points can be applied to object oriented software. We presented a methodology for estimating the size and effort of object oriented software. The method is based on an adaptation of traditional function points to object oriented paradigm. Mapping from FP concepts to OO concepts have been defined, and the OOFPs counting process has been described. The
Adapting Function Points to Object Oriented Information Systems
73
OOFP Counter tools has been developed to automate the counting process. Results obtained from a pilot study in an industrial environment have been reported. The results for size estimation are encouraging, and they can be used with many effort estimation models. Future work will investigate the effect of recalibrating the complexity tables and analyzing the statistical correlation between the collected measeres (DETs, RETs, FTRs) and program size. Other relationships, beyond just OOFP and code size, will be studied; those between OOFP and traditional FP, and OOFP versus effort, are of particular interest.
7
Acknowledgement
The authors are indebted with Cesare Furlanello who performed most of the statistical analysis in the pilot study.
References 1. A. J. Albrecht. Measuring application development productivity. In Proc. IBM Applications Development Symposium, pages 83–92. IBM, Oct. 1979. 2. G. Caldiera, C. Lokan, G. Antoniol, R. Fiutem, S. Curtis, G. L. Commare, and E. Mambella. Estimating Size and Effort for Object Oriented Systems. In Proc. 4th Australian Conference on Software Metrics, 1997. 3. T. Fetcke, A. Abran, and T.-H. Nguyen. Mapping the OO-Jacobson approach to function point analysis. In Proc. IFPUG 1997 Spring Conference, pages 134–142. IFPUG, Apr. 1997. 4. E. Gamma, R. Helm, R. Johnson, and J. Vlissides. Design Patterns: Elements of Reusable Object Oriented Software. Addison-Wesley, 1995. 5. T. Hastings. Adapting function points to contemporary software systems: A review of proposals. In Proc. 2nd Australian Conference on Software Metrics. Australian Software Metrics Association, 1995. 6. IFPUG. Function Point Counting Practices Manual, Release 4.0. International Function Point Users Group, Westerville, Ohio, 1994. 7. IFPUG. Function Point Counting Practices: Case Study 3 - Object-Oriented Analysis, Object-Oriented Design (Draft). International Function Point Users Group, Westerville, Ohio, 1995. 8. Interactive Development Environments. Software Through Pictures manuals, 1996. ¨ 9. I. Jacobson, M. Christerson, P. Jonsson, and G. Overgaard. Object Oriented Software Engineering: A Use Case Driven Approach. Addison-Wesley, 1992. 10. D. Jeffery and J. Stathis. Function point sizing: Structure, validity and applicability. Empirical Software Engineering, 1(1):11–30, 1996. 11. B. Kitchenham and K. K¨ ans¨ al¨ a. Inter-item correlations among function points. In Proc. 15th International Conference on Software Engineering, pages 477–480. IEEE, May 1993. 12. B. Kitchenham, S. Pfleeger, and N. Fenton. Towards a framework for software measurement validation. IEEE Transactions on Software Engineering, 21(12):929– 944, Dec. 1995.
74
G. Antoniol, F. Calzolari, L. Cristoforetti, R. Fiutem and G. Caldiera
13. Rational Software Corporation. Unified Modeling Language, Version 1.0, 1997. 14. J. Rumbaugh, M. Blaha, W. Premerlani, F. Eddy, and W. Lorensen. Object Oriented Modelling and Design. Prentice-Hall, 1991. 15. M. Schooneveldt. Measuring the size of object oriented systems. In Proc. 2nd Australian Conference on Software Metrics. Australian Software Metrics Association, 1995. 16. H. Sneed. Estimating the Costs of Object-Oriented Software. In Proceedings of Software Cost Estimation Seminar, 1995. 17. J. Verner, G. Tate, B. Jackson, and R. Hayward. Technology dependence in Function Point Analysis: a case study and critical review. In Proc. 11th International Conference on Software Engineering, pages 375–382. IEEE, 1989. 18. S. Vicinanza, T. Mukhopadhyay, and M. Prietula. Software-effort estimation: an exploratory study of expert performance. Information Systems Research, 2(4):243– 262, Dec. 1991. 19. S. Whitmire. Applying function points to object-oriented software models. In Software Engineering Productivity Handbook, pages 229–244. McGraw-Hill, 1993.
Global Cache Management for Multi-Class Workloads in Data Warehouses Shudong Jin1, and Xiaowei Sun2 1
Department of Computer Science, Huazhong University of Science & Technology, Wuhan, Hubei 430074, China [email protected] 2 College of Computer Science, Northeastern University, Boston, MA 02115, USA [email protected]
Abstract. Data warehouses usually rely on the underlying database management systems, and operations in warehouses require adequate global cache management for both base data and derived sets. However, a problem is how to ensure the cache balance between the query operations and the retrieved sets of query templates. In this paper, we describe an efficient global cache manager that caches both database pages and derived data. A benefit metric is developed to compare the expected effect of caching retrieved sets and buffering for query processing. On the basis of this model, algorithms are proposed to generate efficient cache allocation scheme for multiple queries. These algorithms can be combined with different replacement strategies. We designed a mixed workload, and made experiments to investigate the performances of the algorithms. The results indicate that our approaches are better than traditional LRU and LRU-K methods for multi-class workloads in data warehouses.
1
Introduction
Warehousing is a promising technique for retrieval and integration of data from distributed, autonomous and possibly heterogeneous information sources [19]. Data warehouses are usually dedicated to the online analytical processing (OLAP) and decision support system (DSS) applications. Many advanced techniques have been incorporated in warehouses to gain an acceptable level of performance. Literature focused mainly on materialized view selection [2, 10] and maintenance [9, 12, 17], and query equivalence for using the views [8]. Only a few researchers have discussed cache management in warehouses. Design of efficient buffer management algorithms has gained a lot of attention. The LRU-K replacement algorithm [15] uses the last K reference times of each cached object. For multiple-class workloads, DBMIN [3] is a classic method. In [14] a flexible allocation method was proposed. It considers different query access patB. Pernici and C. Thanos (Eds.): CAiSE’98, LNCS 1413, pp. 77-91, 1998. Copyright Springer-Verlag Berlin Heidelberg 1998
78
Shudong Jin and Xiaowei Sun
terns and uses adaptable replacement strategies. Recently, Brown et al. revisited this problem [1]. They developed a new method called Class Fencing based on the concept of hit rate concavity. These approaches depend on the uniform size of all pages and the uniform cost of fetching these pages. In data warehouses, the cache criteria should consider different retrieved set sizes, execution costs of associated query templates, as well as their reference rates. In [16], Sellis studied cache replacement algorithms in the context of caching retrieved sets of queries. Several cache replacement strategies were proposed, which sort the retrieved sets by one of above three factors or their weighted sum. Another project ADMS [5] benefits from caching at multiple levels, i.e., the retrieved sets and the pointers to the retrieved records. A recent important work was reported in [18]. Their cache manager aims at minimizing query response time. Two complementary cache replacement and admission algorithms make use of a profit metric. This metric combines the reference rate and size of each retrieved set, and the associated query execution cost. Experimental results indicated that their algorithms outperform conventional LRU replacement algorithm in decision support environments. Unfortunately, caching methods for multi-class workloads in data warehouses have not been explored sufficiently. Especially there exists a problem: How to cache for both the query operations and the retrieved sets of query templates? When the memory contains derived data, and other operations also ask for buffer space, how to ensure a balance between these two needs and achieve maximum gain? It is an important and interesting theme for several reasons. First, many warehouses are constructed on the top of databases, so it’s required to cache both database pages and derived data in a global buffer pool. Second, operations in warehouses can be very complex and multiform: some operations on set data are completed by warehouse software; others may rely on the underlying DBMS and base data. These operations all require memory for processing. Finally, warehouses should support multi-user environments. OLAP/DSS processing and small updates and queries generate mixed workloads. Even a single complex query can be decomposed into several basic operations. In this paper, we investigate global cache management for multi-query workloads in warehousing environments. A practical metric is developed to compare the benefits of caching retrieved sets and buffering for query instances. Algorithms are designed to obtain efficient cache allocation based on this metric. Hence they produce cache allocation schemes considering both needs. These algorithms integrate with different replacement strategies to generate several cache management methods. Simulated results show that our approaches outperform LRU and LRU-K algorithms. The rest of this paper is organized as follows. Section 2 contains a brief introduction to our global cache manager in a warehousing architecture. Section 3 models the profit of caching retrieved sets and the benefit of buffering for query operations, and links them with a practically comparable metric. Then in section 4, we describe the cache management algorithms. In section 5, a mixed workload is designed on an experimental database to study the algorithms’ performances, followed by a brief conclusion.
Global Cache Management for Multi-class Workloads in Data Warehouses
2
79
Architecture
Data warehouses usually rely on traditional DBMS, not only for the data provided by databases, but also for the software construction. For example in warehousing environments, the buffering subsystem should cache both database and warehouse data in a global buffer pool. Figure 1 gives a simple framework of data warehouse management system (DWMS) on the top of a database management system (DBMS), and a global cache manager (GCM) in it. This architecture is similar to our warehouse research project based on a client-server database system. Applications DWMS
Query instances (3)
DBMS
(4)
(1)
Cache the base data
(2)
Cache the derived data
(3)
Access and process the
Cache manager (1)
(2) Derived views
base data (4)
Access and process the derived data
Tables, Indexes, etc
Figure 1. Global cache manager in A DWMS
The global cache manager caches database pages and derived data of warehouses. At low level, derived sets are cached in a granule of individual pages, but each page has an identifier to indicate its corresponding set. The cache is organized efficiently by using hashed lists. Each buffer page contains a page identifier, page content, four link pointers (two for the hashed list and two for the global list), some flags, referenced bits (each bit indicates whether the page is referenced by the corresponding thread), and the last K reference times for LRU-K replacement algorithm. Query threads are called query instances; each is dispatched for a basic operation. Multiple users’ queries, on either base or derived data, can be processed concurrently. Each might be decomposed into several operations. For example, a query can be decomposed into three instances: One retrieves a derived set; one selects records from a table; and the third joins above two results. They call the cache allocation algorithms to require buffer space. After obtaining cache, they can decide how to utilize it. A query instance’s allocated cache is called its private cache. The remains are free to be dedicated to new instances or swapped out by replacement algorithms.
3
Benefit Model
Warehouses are required to cache the retrieved sets of predefined query templates, as well as to provide working area to query operations. In this section we model the
80
Shudong Jin and Xiaowei Sun
profit of caching derived sets and the effect of buffering for query operations, then solve the problem: How to compare the benefits of satisfying these two needs. 3.1 Caching Retrieved Sets Retrieved sets have different accessed frequencies. For example, some small statistical sets (averages, sums, counts, etc.) are usually most frequently referenced, but a multi-attribute projection view seems to be seldom used. To cache retrieved sets of frequently cited templates is more efficient, since they are likely to be used again in the near future. We must have some metric to judge the profit of caching them. In buffer management, the usual criterion for deciding to keep which stored objects should consider their reference probabilities in the future. For future reference patterns are absent in advance, the future reference probabilities are approximated from past reference patterns. This method performs well if the patterns are stable. Let DS1, DS2, ..., DSm be the derived sets of query templates M1, M2, ..., Mm. We use the following statistics of each derived set DSi and associated query template Mi: Fi : average reference frequency of Mi ; Si : size of DSi (in pages) produced by executing Mi ; Ci : average execution cost of Mi . Let us define the benefit of caching DSi. When DSi is not in cache and some query references Mi, DSi will have to be retrieved. (If DSi is not permanently stored on the disks after the last execution of Mi, then the template will have to be recomputed; if DSi is maintained, then the system need only to read the set.) Let Ci denote the retrieval cost, which is assumed able to compute. Therefore, if DSi resides in memory, cost Ci can be saved. We have the following equation to express the average cost saving (it’s so expressed since the smaller sets with higher reference rates and retrieval costs have higher profits): Profit (Mi ) =
Fi × Ci Si
(1)
Obviously, it can benefit from caching the derived sets with high Profit values. To achieve this we should decide the values of Fi, Ci, and Si for each template Mi. As described above, Ci and Si can be determined when Mi is computed or the set is read from the disks. However, computation of Fi is difficult since it depends upon dynamic historical information sampled from processing. As in [18], a similar LRU-K method for set data will be eligible for this work. It records the last K reference times, computes and maintains the reference rate of each retrieved set. 3.2 Caching for Query Instances Queries in data warehouses can be very complex. Some queries are transformed into operations on the derived sets, and others may be directly delivered to the underlying database systems. Even those queries interpreted by warehouse software also
Global Cache Management for Multi-class Workloads in Data Warehouses
81
depend on basic database operations. A complex warehouse query usually contains several operations on base relations; further processing of retrieved sets is also possible, for example joining two views and aggregating the result. Besides in fact, recomputation of the derived sets is performed by query instances. These operations depend on the underlying routines, including the buffering subsystem. The global cache manager should allocate cache as working areas of multiple queries on different data resources. Consider the problem of finding the optimal cache allocation for multiple queries with maximum buffering effect. The standard to measure the effect of buffer management is the disk I/O times. So we consider the expected page faults of the queries. Let Q1, Q2, ..., Qq be the query instances in the system at some time. A query’s access paths determine its reference patterns, and then the number of page faults if buffer size fixed. Thus the number of Qi’s page fault can be defined as a function PFi (Bi), whose only parameter Bi is the buffer number allocated to the query. The objective of cache allocation is to minimize the total page faults ∑iq=1 PFi ( Bi ) under cache size restriction ∑iq=1 Bi ≤ B , where B is the total buffer page number. It says that cache allocation should have greatest effect, or any addition to a query’s cache should reduce its page faults significantly. Here we introduce a metric Eff to measure the effect of cache allocation. It is the average I/O cost saving of a query divided by its occupied cache size. (Note, we will discuss in section 3.3 that our objective is to develop a comparable metric between caching derived sets and buffering for query operations, and the profit metric in section 3.1 is expressed as the cost saving divided by the set size, so we adopt this Eff metric.) The following equation expresses the effect if Bi buffer pages are allocated to Qi: Eff (Qi , B min , Bi ) =
( PFi ( B min ) − PFi ( B i )) × t io Bi − B min
(2)
Here Bi>Bmin, Bmin is the minimum cache size that Qi requires necessary for its processing, and tio is the disk I/O cost of a page. If we hope to allot extra bi>0 buffer pages to Qi, then this addition’s effect is: Eff (Qi , Bi , Bi + bi ) =
( PFi ( Bi ) − PFi ( Bi + bi )) × tio bi
(3)
According to this equation, we can compute the effects of cache addition to the queries and allocate buffer pages to the one with greatest effect. Similarly in [1], Class Fencing estimates the buffer hit rate by exploiting a hit rate concavity assumption. This assumption says that the slope of hit rate curve never increases as more memory is added to an optimal buffer replacement policy. The slope of hit rate curve represents the marginal increase in hit rate curve obtained by adding an additional buffer page. This theorem supports our measures. A problem in above approach is the derivation of the page fault functions. They rely on the access methods of particular queries, and should be declared or estimated by the query processing routines. We give some examples to illustrate them. Let
82
Shudong Jin and Xiaowei Sun
be the page reference sequence. In relational databases, many complex reference patterns are constructed on the basis of three simple patterns: • Sequential reference. Its characteristic is, for ∀i, j, if 1≤i, j≤k and i≠j, then Pi≠Pj. • Random reference. Pi can be any possible page with equal chance, and all Pi’s are independent. • Looping reference. There exists a cycle t, for ∀i, j, if 1≤i, j≤t and i≠j, then Pi≠Pj, but if 1≤i≤k-t, then Pi+t =Pi. As described in [6], we can obtain the expected page fault times of above access patterns as follows, respectively. We can also determine the minimum buffer requirements according to MG-x-y [14]: PFsequential(b) = k
(4)
Bmin,sequential = 1
(5)
N −1 k ln(1 − b / N ) ) ), k < k0 = N ln(1 − 1/ N ) , PFrandom(b) = b + (k − k0 ) × (1 − b / N ), k ≥ k0
(6)
Bmin,random = 1
(7)
N × (1 − (
{
(t − b) × t × (k / t − 1) , t −1 PFlooping (b) = t, b > t t+
{
Bmin,looping = x%×t
b ≤ t,
(8)
(9)
Here b is the allotted buffer number, N is the relation size in pages, and x is an adaptable parameter in MG-x-y family. 3.3 Comparable Metric When query instances execute, the cached derived sets may be swapped out due to competition. It becomes serious if many instances run concurrently and require large amount of cache. So it is important to decide how to use memory, i.e., cache some derived sets for possible future reference to the query templates, or dedicate the cache to current query instances to reduce their page faults. Unfortunately, equation (1) and (3) provide different metrics to measure the potential benefits of caching derived sets and buffering for query instances. It is necessary to develop an approach linking them, thus to compare the benefits of satisfying both needs. Here we give a practical approximation. Let ∆t be the length of an elapsed period, ∆IO be the total page I/O operations completed during this period. Assume that Qi has performed IOi page reads, which is
Global Cache Management for Multi-class Workloads in Data Warehouses
83
monitored and maintained by the cache manager. We obtain the expected query runtime based on historical disk access frequency ∆t/∆IO. The expected remaining I/O times is ∑iq=1 ( PFi ( Bi ) − IOi ) , and the expected runtime is: ET = ∑i =1 ( PFi ( Bi ) − IOi ) × q
∆t ∆IO
(10)
If DSi will be cached during the future ET period, instead of dedicating this portion of buffer to query instances, then the expected cost saving is: EProfit (Mi) = ET × Profit (Mi)
(11)
To decide whether to cache derived sets or to allot buffer to query instances, need only compare the values of equation (3) and (11). The former is used to compute the effect of increasing buffer size for Qi, and the latter estimates that of caching DSi. Above approximated ET may be somewhat inaccurate, but to obtain an accurate ET is considerably complicated, which must consider the processing of all queries. Besides, the approximation also contributes to other factors such as the estimated PFi, and unstable reference patterns, etc. We should further consider a special case in estimating ET. The computation of equation (11) relies on the statistics maintained by the cache manager, but if there is no any query instance in running, then ET can not be computed. If so, when a new query instance Q1 arises, we estimate ET as: ET = PF1 (B1) × EFio
(12)
Here EFio is an approximated I/O frequency in processing, 0≤EFio≤1, and PF1(B1) is the page fault times of Q1 executing with total B1 buffer pages. EFio can be a fixed value or a value hinted by the query processing routines with consideration of access characteristics. For example, a simple scan query’s overload is I/O-intensive, so EFio is close to 1.0, and if the query is CPU-intensive then EFio is lower.
4
Cache Management Algorithms
In this section, we describe the chief algorithms in the global cache manager. The first is greedy algorithm GA, which tries to find the best cache allocation scheme (CAS) for a multiple query workload and the derived sets. Other allocation algorithms base upon it. As an example, algorithm CA for allocating cache to new queries is depicted. We also briefly discuss the buffer admission and several possible replacement strategies. 4.1 Cache Allocation Basic Greedy Algorithm The first greedy algorithm (described below) considers allocating total B buffers to the query instances and query templates. Based on the comparable metric described
84
Shudong Jin and Xiaowei Sun
in section 3, it tries to find an adequate scheme, which achieves maximum benefit and ensures cache balance between the derived data and the query instances. The query templates can be regarded as constant queries, so that they and the query instances can be treated in a uniform way. The initial cache allocation scheme is (B1, B2, ..., Bq, D1, D2, ..., Dm). It means that Bi buffer pages are allocated to each Qi, and Di pages are used to cache the retrieved set of each Mi. It always satisfies ∑iq=1 Bi + ∑im=1 Di ≤ B , the cache size constraint. Di can be Si or 0, representing whether to intend caching the derived set. But Di=Si does not mean that DSi must exactly be cached, and Di=0 does not mean that DSi is not in memory currently. In fact a derived set can be partially cached. Algorithm GA: The greedy algorithm trying to find the best allocation scheme. Input: CAS=(B1, B2, ..., Bq, D1, D2, ..., Dm); ’ ’ ’ ’ ’ ’ ’ ’ Output: new CAS=(B 1, B 2, ..., B q, D 1, D 2, ..., D m), in which B i≥Bi, D i=Si or 0, ' ' ∑iq=1 Bi + ∑im=1 Di ≤ B ; BEGIN ’ // Queries only occupy previous size (1) FOR i=1 TO q DO B i=Bi; ’ // No set is proposed to be cached first (2) FOR i=1 TO m DO D i=0; (3) AvailB = B − ( ∑qi=1 Bi' + ∑im=1 Di' ); // The total free buffer number (4) REPEAT (5) FOR i=1 TO q DO // Compute effects of adding query buffer (6) Effect[i] = maxb=1, …, AvailB ( Eff(Qi, Bi, b) ); (7) MaxBi = the b with maximum cost saving; (8) FOR i=1 TO m DO // Estimate effects of caching derived sets ’ (9) IF (D i ≠ 0 or Si >AvailB) THEN Effect[q+i] = 0 (10) ELSE Effect[q+i] = ET×Profit(Mi); // Then execute the most effective cache allocation (11) IF (some Mi has maximum effect) THEN ’ (12) {D i = Si ; AvailB = AvailB − Si } (13) ELSE (some Qi has maximum effect) ’ ’ (14) {B i=B i + MaxBi; AvailB = AvailB − MaxBi }; (15) UNTIL (AvialB==0 or maximum effect==0); ’ ’ ’ ’ ’ ’ (16) RETURN (B 1, B 2, ..., B q, D 1, D 2, ..., D m); END The algorithm takes the initial allocation scheme as it input and produces a new ’ ’ ’ ’ ’ ’ ’ scheme, (B 1, B 2, ..., B q, D 1, D 2, ..., D m). It satisfies B i≥Bi, since that once the buffer pages have been provided to Qi, it will have been using them, so the pages can not ’ be deprived from it before it terminates. But D i can change from Si, to 0, which ’ means that the cached DSi is to be evicted out according to the new scheme. If D i changes from 0 to Si, then the new scheme advises to cache DSi. The algorithm is written in pseudo code. It’s a repetitive procedure. Each loop finds the most beneficial addition, i.e., cache some derived set or increase a query’s
Global Cache Management for Multi-class Workloads in Data Warehouses
85
buffer size. This repetitive procedure terminates if no buffer can be allocated, or no cost saving can be gained. Its major cost is that of cost saving computation, especially judging the most effective buffer size for Qi in line (6), which attempts every possible buffer number. An improvement is to adopt a binary search, which may lead to sub-optimal allocation scheme. In fact, with the hit rate concavity assumption in [1], we can simplify this search. It need only compute the maximum effect of one-page buffer addition to all query instances. Above greedy algorithm is the kernel of cache allocation. Other algorithms rely on it, e.g., CA for allocating cache to new queries, which will be discussed below. Another allocation algorithm EA in our design is used to extend the query’s buffer space for more efficient processing. This algorithm is useful since some instances will terminate and release their occupied cache. It decides a requesting query instance’s extended cache size by also calling GA to obtain the current best scheme. Cache Allocation for New Queries On the basis of the greedy algorithm, the rather simple algorithm CA is described below. It re-computes the allocation scheme when a new query requires cache. The output of CA is the query instance’s allocated cache size, and other query instances’ ’ cache sizes are not adjusted. However, all D i’s can be different to their previous values. It indicates that, some derived sets should be evicted out of the cache to support more efficient query processing, including that of the new query. Algorithm CA: The algorithm allocating cache to new queries Input: CAS=(B1, B2, ..., Bq, D1, D2, ..., Dm), new query Qq+1, its minimum cache requirement Bmin and page fault function PFq+1; ’ ’ ’ ’ Output: CAS=(B1, B2, ..., Bq, B q+1, D 1, D 2, ..., D m) and return Bq+1 if allocate successfully; otherwise no update to CAS and report failure; BEGIN (1) IF Bmin > B − ∑qi=1 Bi THEN Report no available buffer and terminate; ’ ’ ’ ’ ’ ’ (2) (B 1, B 2, ..., B q, B q+1, D 1, ..., D m ) = GA( (B1, B2, ..., Bq, Bmin, 0, ...,0) ); ’ ’ ’ ’ (3) CAS = (B1, B2, ..., Bq, B q+1, D 1, D 2, ..., D m); ’ (4) RETURN (B q+1); END If current available cache can afford the minimum requirement of query Qq+1, algorithm CA will call GA. The input of GA is the minimum fixed cache sizes occupied by the query instances, i.e., Bi for Qi (i=1, …, q) and Bmin for Qq+1. The execution ’ ’ of GA outputs a new scheme, where each B i or D i can differ to previous Bi or Di. ’ After that CA generates a new scheme: B q+1 buffer pages are allocated to the new ’ query; other query instances maintain previous cache sizes; and the updated D i are included. This new scheme reflects the needs of current workload. Consider the reasons why the new scheme does not combine the new cache sizes for other query instances. First, once a query instance has occupied some cache and
86
Shudong Jin and Xiaowei Sun
is processing, it usually can not use additional buffer since its access method only utilizes the original cache. Thus it is wasteful to allocate the available buffer pages to it because they can be dedicated to possible new queries. Furthermore, we should note, these buffer pages, even if not allocated, are still in use as the free portion of the global cache pool.
4.2 Cache Admission and Replacement The buffer scheduler is an internal algorithm of the global cache manager. When too many queries are in the system, they may contest the limited buffer pages. Some might be blocked if current available buffer pages are not enough to meet their minimum requirements. When a query completes, its buffer page number will be returned to the cache manager. At this point the scheduler begins to work. It selects one or more buffer-waiting instances according to some criteria, and wakes up them. It also decides adequate cache sizes for these queries. For this, it adds new elements into the cache allocation scheme and calls algorithm GA. In our method, replacement strategies are necessary at two levels. The first level, private cache replacement, means that the query instances can adopt efficient replacement strategies fitting their access patterns. The query optimizer and processing program, instead of our global cache manager determine this level, so it is not this paper’s main topic. The other level is global cache replacement. It seems that this level is unimportant since the available cache is allotted to the query instances as private buffer pages. Contrary, it has great effect on the performance since: • Not all buffer pages serve for the query instances. The global cache, excluding the allocated parts, can be a large portion of the buffer pool. Many pages cache DSi’s. Buffer sharing also causes the free portion larger than expected. • Different types of free buffer pages have unequal values of being kept in cache. Some are the query instances’ wasted pages; others are used to keep the derived sets. They require adequate replacement strategies. A simple cache replacement strategy is LRU, which is a widely studied and adopted approach in OS and database system fields. As pointed out in [15], this replacement algorithm performs inadequately in the presence of multiple workload classes, due to the limited reference time information it uses in the selection of replacement victims. Consequently, the LRU-K algorithm has been proposed, which considers the times of the last K≥1 references to each page. We can also adopt this method. These two methods have a side effect. They are page-oriented, or they swap out the buffer pages in individual pages. Hence, usually a derived set is partially swapped out of the cache though the allocation scheme proposes to cache it. Some derived sets might be wholly evicted out of memory. We simply allow this inconsistency between the scheme and the replacement algorithm’s actual results. We can also adopt another strategy called CAS-prior. Contrary to the LRU-K’s ignoring the allocation scheme, this strategy aims to keep the derived sets with higher benefits prior to other database pages and derived sets. According to this
Global Cache Management for Multi-class Workloads in Data Warehouses
87
strategy, if in the scheme Di=Si, then DSi should still in cache if it resides currently. Replacement algorithm only considers other individual pages and set data. From these candidates, the replacement algorithm can use LRU-K strategy for selection. The CAS-prior strategy does not rule that the derived sets declared in the allocation scheme are exactly those cached. A derived set proposed in cache can be out of memory if it has not been referenced recently. On the other hand, other data can also be in cache. When a query terminates, its buffer pages will still cache its used data before they are evicted out. This query can also have referenced derived sets that are not suggested to reside in memory.
5
Experimental Study
5.1 Experimental Environment We made experiments on the top of a client-server database management system. The system ran on a SGI Indigo 2 workstation. We loaded a medium size database and derived data, designed a multi-class workload stream to verify the performance of the global cache manager. A database of about 140M bytes was created. It contains 1,000,000 records, and + each record is 100 bytes in size. The database is organized using a B -tree index. All data and index page size is 4K bytes, the same as a buffer page. Table 1. Experimental Derived Sets
We also defined 10 query templates and stored the data sets in the disks. These query templates have different sizes and reference rates. Table 1 gives their storage sizes, reference rates, and retrieval costs. It is designed that the smaller views have greater probabilities to be referenced, but their retrieval costs are proportional to the sizes. The total size of the derived sets is 4,000 pages. We assumed the costs in Table 1 contribute mainly to the I/O costs, since we designed that the derived sets were stored in the disks and would be retrieved by scanning them. Thus unlike [18], where cost saving is the main metric, we simplify the problem and the hit rate becomes a reasonable performance metric.
88
Shudong Jin and Xiaowei Sun
5.2 Workload Design We designed a multi-class workload on both database and derived data. Table 2 gives the frequencies in QPS (Queries Per Second) of all five kinds of queries. Two kinds of simple queries on the base database are: • Q1: Exact match query. It searches only one record through the B+-tree by matching a randomly generated key value. • Q2: Range query. It scans a database segment (its range is also randomly decided), accessing 100 data pages. We also designed other three kinds of queries on the derived data. These queries can be translated into operations to the sets. They stochastically pick some derived sets. The sets’ probabilities of being selected are proportional to their reference rates in Table 1. These three types of queries are: • Q3: Simple scan query, which scans a randomly selected set. • Q4: Sort-based aggregate-query of a randomly selected set. • Q5: Sort-merge join of two randomly selected sets. We mixed these queries in a query stream, and ran many versions of this stream simultaneously (Note, these versions are not simple duplications but actually different streams due to the random number generator). To ensure good effects, we ran as many versions as possible, e.g., up to 30 or 50 if the available system resources permit. Each version’s query frequency can be computed through dividing 14.042 QPS by the number of concurrent versions. Table 2. Query Frequencies
Queries QPS
Q1 10
Q2 0.2
Q3 3
Q4 0.8
Q5 0.042
5.3 Performance Metric We used the primary performance metric page hit rate (HR) during some processing period. It’s defined as: HR =
TotalPageHit TotalPageReference
(13)
We converted the hit rate of derived sets into the uniform hit rate of individual pages. The cache manager monitors I/O operations and page references. Finally, they are summed up to compute HR. To obtain more accurate results, we ran the query streams for a long period from 20 minutes to an hour. The parameters are sampled from hot system when the cache manager has become experienced. Three GCM-based approaches are compared with other two non-GCM-based approaches. The latter two are LRU and LRU-K (K=3) without cache allocation mechanism. All query instances fully share and compete for the global cache. On the other hand, three GCM-based approaches are (detailed description in section 4):
Global Cache Management for Multi-class Workloads in Data Warehouses
89
• GCM-LRU, where the global cache manager uses page LRU replacement for the free portion of the buffer pool. • GCM-LRU-K, where the global cache manager uses page LRU-K replacement for the free portion. • GCM-CAS-prior, where the global cache manager uses CAS-prior replacement for the free portion. 0.6
HR
0.5 0.4 0.3 0.2
LRU LRU-K GCM-LRU GCM-LRU-K GCM-CAS-prior
0.1 0 1
2
4
8
12
16
Cache size (M bytes)
Figure 2. Hit rates of the algorithms
5.4 Experimental Results Figure 2 gives the experimental results. The hit rates of simple LRU are always the worst for any cache size, and they are well below the others. Although the LRU-K method has not adopted adequate cache allocation for multiple queries, we find that LRU-K strategy’s hit rates are almost two times high as those of LRU. Obviously LRU-K benefits from more historical information than LRU. A larger K improves the estimation of the individual page reference rates, consequently leads to a higher HR. We should point out that a larger K could play a more significant role under multi-class workloads in which each class has different reference characteristics. GCM-LRU approach has acceptable hit rates. It implements cache allocation for multiple queries and enables flexible cache replacement. But sometimes it is worse than the LRU-K strategy. This proves again the high performance of LRU-K. Another phenomenon is, when the cache size is small, its hit rates are obviously lower than LRU-K’s, but if the buffer pool is enlarged, GCM-LRU closes, even outperforms the latter. Compared with GCM-LRU, GCM-LRU-K has consistently higher hit rates for all cache sizes. We find that it is obviously better than the simple LRUK strategy. The principal reason is, it combines efficient cache allocation for multi-
90
Shudong Jin and Xiaowei Sun
ple queries and the experienced LRU-K replacement. Each query instance has its private cache replacement adaptable to its reference pattern. However, the most successful one is GCM-CAS-prior. It is slightly better than GCM-LRU-K. In GCM-LRU or GCM-LRU-K, the derived sets are possible to be evicted out of the global cache wholly or partly ignoring the cache allocation algorithms’ propositions. This phenomenon becomes more possible when some costly derived sets are not repetitively referenced. Contrary, GCM-CAS-prior adopts effect estimation and caches the most beneficial data, and the replacement algorithm retains the sets declared in current allocation scheme. In above experiments, we assumed that the execution overhead of query templates principally contributes to the retrieval I/O cost. In fact the computation of small sets can also be costly. If so, our GCM-CAS-prior approach will be more evidently superior to GCM-LRU-K and the others. We have not experimented to compare our method with that of [18] partly because we have not implemented their algorithms. Their performance analysis was also strong with respect to queries and data, but they had not considered buffer requirement of query instances. Actually when other operations ask for large amount of cache, the performance of caching derived sets will be affected significantly.
6
Conclusion
This paper investigates the problem of global cache management for multiple queries in data warehouses. A practical comparable profit model is developed to compare the potential benefits of caching retrieved sets and buffering for query operations. Algorithms are designed to gain efficient cache allocation schemes, which arbitrate the balance between above two needs. Together with different replacement strategies, these algorithms generate several approaches. Experimental results indicate that our methods have better performances than simple LRU and LRU-K strategy. Researches on the following topics should be further investigated: • Effects of warehouse modifications. In this paper we have not considered their possible effects on cache management in warehouses, though updates are not frequent in such environments. • Load control problem. Our global cache manager provides some load control mechanism. Each query instance requires a minimum cache size, so concurrent query number is limited. But it is not enough yet. • Benchmark performance evaluation. We hope to make experiments based on TPC-D or Set Query benchmarks.
References 1.
Brown, K.P., Carey, M.J., Livny, M.: Goal-oriented buffer management revisited. In Proceedings of ACM SIGMOD Conference, 1996: 353-364
Global Cache Management for Multi-class Workloads in Data Warehouses
91
2.
Baralis, E., Paraboschi, S., Teniente, E.: Materialized view selection in a multidimensional database. In Proceedings of VLDB Conference, 1997: 156-165
3.
Chou, H., DeWitt, D.J.: An evaluation of buffer management strategies for relational database systems. In Proceedings of VLDB Conference, 1985: 127-141
4.
Chan, C., Ooi, B., Lu, J.: Extensible buffer management of indexes. In Proceedings of VLDB Conference, 1992: 444-454
5.
Chen, C., Roussopoulos, N.: The implementation and performance evaluation of the ADMS query optimizer: Integrating query result caching and matching. In Proceedings of EDBT Conference, 1994: 323-336
6.
Faloutsos, C., Ng, R., Sellis, T.: Predictive load control for flexible buffer allocation. In Proceedings of VLDB Conference, 1991: 265-274
7.
Faloutsos, C., Ng, R., Sellis, T.: Flexible and adaptable buffer management techniques for database management systems. IEEE Trans. on Computers, 44(4), 1995: 546-560
8.
Gupta, A., Harinarayan, V., Quass, D.: Aggregate-query processing in data warehousing environments. In Proceedings of VLDB Conference, 1995: 358-369
9.
Gupta, A., Mumick, I.: Maintenance of materialized views: Problems, techniques, and applications. IEEE Data Engineering Bulletin, 18(2), 1995: 3-18
10. Gupta, A.: Selection of views to materialize in a data warehouse. In Proceedings of ICDT Conference, 1997: 98-112 11. Harinarayan, V., Rajaraman, A., Ullman, J.: Implementing data cubes efficiently. In Proceedings of SIGMOD Conference, 1996: 205-216 12. Huyn, N.: Multiple-view self-maintenance in data warehousing environments. In Proceedings of VLDB Conference, 1997: 26-35 13. Mohan, N.: DWMS: Data warehouse management system. In Proceedings of VLDB Conference, 1996: 588 14. Ng, R., Faloutsos, C., Sellis, T.: Flexible buffer allocation based on marginal gains. In Proceedings of SIGMOD Conference, 1991: 387-396 15. O’Neil, E., O’Neil, P., Weikum, G.: The LRU-K page replacement algorithm for database disk buffering. In Proceedings of SIGMOD Conference, 1993: 297-306 16. Sellis, T.: Intelligent caching and indexing techniques for relational database systems. Information Systems, 13(2), 1988: 175-185 17. Staudt, M., Jarke, M.: Incremental maintenance of externally materialized views, In Proceedings of VLDB Conference, 1996: 75-86 18. Scheuermann, P., Shim, J., Vingralek, R.: WATCHMAN: A data warehouse intelligent cache manager. In Proceedings of VLDB Conference, 1996: 51-62 19. Widom, J.: Research problems in data warehousing. In: Proceedings of CIKM Conference, 1995.
OMS/Java : Model Extensibility of OODBMS for Advanced Application Domains Andreas Steiner, Adrian Kobler and Moira C. Norrie {steiner, kobler, norrie}@inf.ethz.ch Institute for Information Systems ETH Zurich, CH-8092 Zurich, Switzerland
Abstract. We show how model extensibility of object-oriented data management systems can be achieved through the combination of a highlevel core object data model and an architecture designed with model extensibility in mind. The resulting system, OMS/Java, is both a general data management system and a framework for the development of advanced database application systems. All aspects of the core model – constructs, query language and constraints – can easily be generalised to support, for example, the management of temporal, spatial and versioned data. Specifically, we show how the framework was used to extend the core system to a temporal object-oriented database management system.
1
Introduction
Extensibility has often been considered a purely architectural issue in database management systems (DBMS). In the 1980s, there was an increase in the various forms of DBMS that appeared — many of which were tailored to specific application domains such as Geographical Information Systems or Computer Aided Design systems. It was recognised that no single DBMS could hope to satisfy the requirements of all such applications and the idea of an extensible DBMS emerged. Thereafter, a number of extensible systems appeared – including kernel e.g. [CDKK85], customisable e.g. [HCL+ 90], toolkit e.g. [CDG+ 89] and generator e.g. [MBH+ ] systems. In these systems, the emphasis tends towards providing architectural support in terms of how to engineer new DBMS from existing frameworks and services such as storage management, transaction management, access methods and query optimisation and we therefore call this architecture extensibility. The idea evolved of a database engineer as someone whose task would be to construct an advanced DBMS based on such a system – and, frequently, this was a significant task. A discussion of general requirements of such systems is given in [GD94]. In contrast, there have been proposals for various forms of model extensibility in which new types and/or operators can be introduced in a DBMS to meet the needs of specific application domains, e.g. [BLW88,SC94,Cha96]. We aim for a higher-level of model extensibility in which all aspects of a data model – structures, query language and constraints – can be generalised. This ensures B. Pernici and C. Thanos (Eds.): CAiSE’98, LNCS 1413, pp. 115–138, 1998. c Springer-Verlag Berlin Heidelberg 1998
116
Andreas Steiner et al.
upward compatibility in that the core model, languages and mode of working remain essentially the same. For example, generalisation to a temporal system enables temporal semantics to be associated with all constructs, operations and constraints of the core model. Further, non-temporal queries will be unaffected and temporal queries have the same form as non-temporal queries – with only prefixes required to specify that a temporal semantics is required. Similarly, the DBMS could be generalised to handle spatial data, versioned data, multimedia data and any combination of these. OMS/Java is both a framework and an object data management system designed with such model extensibility in mind. The system is based on the OM model which, through a clear separation of model concepts, orthogonality of design and general modelling expressiveness has the properties required of the core model of such an extensible system. The development and architecture of OMS/Java exploited current software engineering principles such as application frameworks and design patterns to ensure that benefits of reusability and extensibility could be fully exploited. In this paper, we describe the OMS/Java architecture and system and how its extensibility was exploited to implement a temporal object-oriented database management system with minimal effort. We begin in Sect. 2 and 3 with a discussion of the general requirements for systems with true model extensibility. This includes a discussion of the requirements of an appropriate core model along with a description of our core model OM. Section 4 then presents the general OMS/Java architecture, followed with a description of the OMS/Java system. To demonstrate the process of model extensibility, we describe the development of the temporal OMS/Java system, TOMS/Java, in Sect. 5. We conclude with a summary and outline of future work in Sect. 6.
2 2.1
Architecture Reusable Parts of a DBMS
Modern DBMS have to support a variety of tasks and requirements [EN94,HS95]. A DBMS manages persistent data, meaning that data exists longer than tasks run by application programs. Data structures are defined to describe and store the data in a uniform way. Sophisticated access methods (for example, using indexing or hashing) provide for efficient data access. (Declarative) languages and operations need to be supported which allow the definition, modification and retrieval of data. The DBMS should also provide means and techniques to optimise data retrieval. Transaction management and concurrency control allow multiple users to access the same data simultaneously, avoiding any problems which might evolve. Additionally, a modern DBMS should support concepts to provide data consistency, recovery from system crashes and database security and authorisation. The implementation of an extensible data model as a DBMS should still support all of these tasks while allowing the user to extend the model and system with new functionality. Currently available DBMS support the features listed
OMS/Java : Model Extensibility of OODBMS
117
above in one way or another. Relational DBMS support a single data structure – the relation – and a non-extensible functionality. Object-oriented DBMS support extensibility of the data structures through the use of type constructors such as tuple, set, bag, list and so on and extensibility of the functionality through the possibility to store methods in the database. It is difficult to efficiently and effectively support application domains which are based on specialised data models – for example, the management of temporal or spatial data or versioning – using such systems. The limitations of the relational data model with respect to data structures and functionality make it necessary to either build the additional constructs needed into the application itself or implement a specialised DBMS. With respect to object-relational or object-oriented DBMS, it is possible to extend the system to support such advanced application domains. We have investigated such an approach in the area of temporal databases [SN97b]. The major drawbacks of this approach, however, are that the DBMS cannot use the special semantics – for example, the temporal semantics – for query optimisation, that the resulting retrieval and update operations turn out to be very complicated and error prone and that the specification of specialised integrity constraints usually may only be supported through the implementation of methods. Our idea of extensibility thus goes a step further. We would like to have a DBMS based on an extensible data model which allows the reuse of features such as persistence, query optimisation, efficient access methods, transaction management, crash recovery and authorisation, while supporting a generalised form of its data structures, algebra operations , query language and integrity constraints with special semantics. Such a system is upwards compatible in the sense that the basic data structures, operations and constraints can always be used, even in more advanced, generalised forms of the underlying data model, and that for each generalisation of the data model, the same concepts and manner of work can be used. The next section discusses in more detail the generalisation approach and introduces an architecture supporting this method of achieving more advanced data models. 2.2
The Generalisation Approach
A data model M can be considered to consist of three parts – the data structures DS, the operations for data retrieval and update OP and the integrity constraints C, M = (DS, OP, C). In [SN97c], we have introduced a temporal data model which – in contrast to other temporal data models – enhances all three parts of a nontemporal data model to support the management of temporal data. Figure 1 shows the three parts of a data model. Most of the temporal data models extend non-temporal data models in that they add special timestamp attributes to store temporal data and add special operations to query temporal data. Other ones extend the data structures but supply a special temporal algebra. Our generalisation approach, however, considers all three parts of the data model by supporting temporal object identifiers, temporal algebra operations and temporal constraints. The resulting temporal object model is independent of any
118
Andreas Steiner et al.
type system and orthogonal in the sense that anything modelled as an object, including collections of objects, constraints and even types, can be timestamped. Data Structures Generalisation of Data Structures, Algebra and Constraints
Constraints
Generalisation of Algebra Generalisation of Algebra and Constraints
Algebra
Fig. 1. The three parts of a data model
A DBMS should additionally provide for a language which can be used to specify the data structures and integrity constraints used for an application (the data definition language), update operations (the data manipulation language) and queries (the query language). This language thus consists of three sublanguages. In order to come up with a DBMS which allows the reuse of the general tasks of query optimisation, efficient access methods, transaction management, crash recovery and so on, but supplies generalised data structures, algebra operations and integrity constraints, the notion of an object identifier, the algebra operations, the integrity constraints and the language supported must be exchangeable, respectively, extendible. Such an architecture is depicted in Fig. 2.
QL DDL
Core System
DML
OID
Algebra
Constraints
Temporal OID
Temporal Algebra
Temporal Constraints
Fig. 2. Another form of extensibility in a DBMS
OMS/Java : Model Extensibility of OODBMS
119
The core system supports the basic tasks of persistence, query optimisation, efficient access methods, transaction management, crash recovery, authorisation and so on. The object identifier, the algebra and constraints can be extended to support the needs of an advanced application domain, for example, temporal databases. In this case, the notion of an object identifier is generalised into a temporal object identifier which – besides a unique reference – contains a lifespan denoting when the object existed, for example with respect to the real world. To be able to query the temporal data stored in the form of temporal objects, the non-temporal algebra operations are overridden with algebra operations having the desired form of temporal semantics. Finally, the integrity constraints are replaced by temporal integrity constraints. The same approach can be used to support data models supporting the management of spatial data or versioning. For spatial data, we exchange, respectively, extend the notion of an object identifier with a spatial identifier, which means that the position of an object or the space that it covers is attached to the object identifier. Algebra operations and integrity constraints are replaced with operations and constraints which are able to deal with the new semantics. For versioning, the version number is attached to the object identifier and again, operations and integrity constraints which are able to deal with the versioned objects are added to the system.
3
Requirements for Model Extensibility
The generalisation approach, as introduced in the last section, obviously deals with objects and algebra operations and integrity constraints referring to these objects. Usually, it is argued that advanced application domains such as the management of temporal, spatial and versioned data can be supported using extensible DBMS which support abstract data types. Methods have to be written to support queries and constraints dealing with the special semantics of the application domain. Clearly, such approaches are based simply on the type system of the DBMS used. The disadvantage of these approaches are, for example, that the special semantics usually cannot be used for query optimisation, and that the specification of queries and integrity constraints in the core model and the advanced model are not similar. Our generalisation approach is based on the object level, which means, that we do not want to deal with instances of types and add additional attributes with special semantics. We want to deal with objects which inherently have special semantics, and a closer look at them reveals their property values. In other words, we advocate an approach which takes the special semantics of an advanced data model such as a temporal data model into account and does not treat temporal properties as just another, but somehow a bit more special, attribute value. Our idea of an extensible data model is that it should support a rich set of algebra operations and constraints which work on collections of objects. The following subsections discuss the requirements for model extensibility in more detail and introduce a data model which supports the generalisation approach.
120
3.1
Andreas Steiner et al.
Separation of Typing and Classification
[Nor95] distinguishes between classifying entities into categories according to general concepts of the application domain and describing the representation of entities within the database in terms of types. For example, we can classify persons as being employees, students or tennis players. Each employee, student and tennis player is a person. Students can further be classified as studying either engineering or mathematics. This is depicted on the left hand side of Fig. 3. On the other hand, we represent persons as having a name and a birthdate, while employees additionally have an employment number and a salary, students have a student number and tennis players have a ranking and are member of a club. This type hierarchy is depicted on the right hand side of Fig. 3. person: Persons
Name: String Birthdate: Date Sub-Collections Sub-Types
Employees
Students
TennisPlayers
employee:
student:
tennisplayer:
EmpID: Integer
StudID: Integer
Ranking: Integer
Salary: Integer
Engineering
Club: String
Math
Fig. 3. Distinguishing typing and classification
In reality, a person may be classified as a student and a tennis player simultaneously. This means, that a person may play the role of an employee and a tennis player at the same time. In order to support the modelling of such facts, a data model must allow objects to have different types simultaneously. In our example, it must be possible that the same person object can be dressed with the types employee and tennisplayer. Most of the object-oriented data models do not support such role modelling. An object is considered to be an instance of a single type. The next subsection introduces a data model which separates typing from classification and supports role modelling. 3.2
The OM Model
Figure 4 shows a simple schema using the notation provided in the OM model. It models collections Persons and Addresses which are related with each other through association have. Cardinality constraints specify that a person has one or more addresses, while an address may be related with one or more persons. Collection Persons has three subcollections, namely Employees, Students and TennisPlayers. The integrity constraint cover demands that each person is also classified in at least one of the subcollections, or, in other words, that collections Employees, Students and TennisPlayers cover collection Persons.
OMS/Java : Model Extensibility of OODBMS
121
In the shaded region of each box denoting a collection, the member type of the collection is given. Collection Persons, for example, has a member type person, which means that each object contained in collection Persons must be dressed with type person. person
[1:*]
address
[1:*] have
Persons
Addresses
cover
employee Employees
student Students
tennisplayer TennisPlayers
Fig. 4. A schema in the OM model
The Data Structures. The OM model supports collections of objects and associations. A collection of objects contains objects dressed with the same member type. A collection is itself an object and thus has an object identifier, and it is possible to have collections of collections. Figure 5 shows how role modelling is supported in the OM model. An object may be classified in two different collections simultaneously, for example in a collection Employees and a collection TennisPlayers. Depending on the collection through which we view this object, we see a different representation of it. Looking at an object through collection Employees shows attribute values such as a name, a birthdate and an employment number plus salary as defined in type employee. Viewing the same object through collection TennisPlayers, however, shows attribute values according to type tennisplayer. Name: Birthdate: EmpID: Salary:
John 1950 13432 10 000
Name: Birthdate: Rank: Club:
Emlpoyees [employee]
John 1950 8 TCR
TennisPlayers [tennisplayer]
OID
Fig. 5. Viewing an object in different roles
An association allows objects in collections to be related with each other. An association is a binary collection since it contains pairs of object identifiers where one object identifier refers to an object in a source collection while the other one refers to an object in a target collection. Associations are objects as well and thus have an object identifier.
122
Andreas Steiner et al.
Algebra. The operational model of OM is based on a generic collection algebra. A list of operations supported in OM is given in Table 1. The algebra includes operations such as union, intersection, difference, cross product, flatten, selection, map and reduce as well as special operations over binary collections such as domain, range, inverse, compose and closure. Table 1. Algebra operations supported in the OM model Algebra Operation Union Intersection Difference Cross Product Flatten Selection Map Reduce Domain Range Inverse Compose Closure
Signature ∪ : (coll[T ype1 ], coll[T ype2 ]) → coll[T ype1 t T ype2 ] ∩ : (coll[T ype1 ], coll[T ype2 ]) → coll[T ype1 u T ype2 ] − : (coll[T ype1 ], coll[T ype2 ]) → coll[T ype1 ] × : (coll[T ype1 ], coll[T ype2 ]) → coll[(T ype1 , T ype2 )] flatten : coll[coll[T ype]] → coll[T ype] σ : (coll[T ype], T ype → boolean) → coll[T ype] map : (coll[T ype1 ], T ype1 → T ype2 ) → coll[T ype2 ] reduce : (coll[T ype1 ], (T ype1 , T ype) → T ype, T ype) → T ype domain : coll[(T ype1 , T ype2 )] → coll[T ype1 ] range : coll[(T ype1 , T ype2 )] → coll[T ype2 ] inv : coll[(T ype1 , T ype2 )] → coll[(T ype2 , T ype1 )] ◦ : (coll[(T ype1 , T ype2 )], coll[(T ype3 , T ype4 )]) → coll[(T ype1 , T ype4 )] closure : coll[(T ype1 , T ype2 )] → coll[(T ype1 , T ype2 )]
The algebra operations are given specifying the input arguments and the results along with their types. Collections which list two types, for example, coll[(T ype1 , T ype2 )] are binary collections, containing objects of the form , where oid1 refers to an object of type T ype1 and oid2 to an object of type T ype2 . For set operations, the notions of least common supertype and greatest common subtype are used to determine the member type of the resulting collections. The union of two collections thus returns a collection whose type is the least common supertype of the two collections involved (T ype1 t T ype2 ). The resulting collection contains all the elements belonging to one (or both) of the argument collections. The intersection of two collections, on the other hand, returns a collection whose type is the greatest common subtype of the two collections involved (T ype1 u T ype2 ), containing the objects which are members of both argument collections. The type of the resulting collection of the difference of two collections corresponds to the type of the first argument in the operation. The resulting collection contains exactly those objects in the first argument collection which are not also members of the second one. The cross product of two collections returns a binary collection coll[(T ype1 , T ype2 )] containing all combinations of an object of the first collection with an object of the second one. The selection operation has as arguments a collection and a function which selects objects from the collection. The result is a subset of
OMS/Java : Model Extensibility of OODBMS
123
the argument collection. The map operator applies a function with a signature T ype1 → T ype2 to each object in the argument collection and returns a collection of objects of type T ype2 . The flatten operator takes a collection of collections of the same member type and flattens them to a collection of type T ype. The reduce operator allows the execution of aggregate functions over a collection of values. It has three arguments – the collection coll[T ype1 ] over which the aggregate function is executed, the aggregate function itself with a signature (T ype1 , T ype) → T ype and an initial value of type T ype. The OM model also supports special operations over binary collections. Some of them are listed in the second part of Table 1. The domain operation takes a binary collection and forms a collection of all the objects that appear as the first element of a pair of objects belonging to the binary collection, whereas the range operation returns a collection containing the second elements of such pairs. The inverse operation swaps the first and the second element of the pairs contained in the argument binary collection and returns them in a new binary collection. The composition operation combines those binary objects of the two argument collections where the second object in the first binary object appears as first object of the second binary object, . The resulting collection contains binary objects, for example, . The closure of a binary collection is the reflexive transitive closure of a relationship represented by the binary collection. Union, intersection and difference of collections, cross product, compose, range, domain, flatten, inverse, closure simply manipulate object identifiers. Exceptions are the selection, map and reduce operations. A selection operation, for example, might access attribute values of the objects in discourse to select the desired ones.
Constraints. The OM model supports constraints such as the subcollection, cover, disjoint, intersection and cardinality constraints. The subcollection constraint demands that each object in a collection is also member in its supercollections. The disjoint, cover and intersection constraints over collections restrict the form of relationship between supercollections and their subcollections. The disjoint constraint demands that a set of subcollections do not share a single object of their common supercollection, whereas the cover constraint demands that each object of the supercollection appears in at least one of the subcollections involved in the constraint. The intersection constraint relates a subcollection with its supercollections such that all the common objects of the supercollections are also members of the subcollection. The cardinality constraints are used to restrict how often an object may be contained in an association. These constraints can be evaluated again by simply manipulating object identifiers. The constraint themselves are also objects.
124
4
Andreas Steiner et al.
OMS/Java
OMS/Java (Object Model System for Java) [Mes97] can be considered as an object-oriented database management system, respectively, as an object-oriented framework for the Java environment supporting the OM generic data model presented in Sect. 3.2. There exist also other instantiations of the OM model for different environments such as OMS/Prolog [NW97] and OMS/Objectivity [Pro97]. OMS/Java can be characterised by three main features: – The OMS/Java core system is extensible in the form described in Sect. 2.2 – OMS/Java can either serve as an Object-Oriented Database Management System (OODBMS) or as an Application Framework. Used as an OODBMS, OMS/Java can be, for example, a server component in a client/server environment. In the other case, a developer can use the OM model to analyse and design the application domain and then use the OMS/Java framework for implementing the applications. Hence, no mapping is necessary between the resulting design model and the implementation. – Using OMS/Java for managing instances of Java classes is easy and does not cause any major redesign of existing class hierarchies. Figure 6 shows a scenario where several client applications are connected to the same OMS/Java database server. Client Application
Client Application
OMS/Java
OMS/Java
…
OMS/Java Server Abstract Database Interface
Objectivity/DB
Sybase
ObjectStore
…
Fig. 6. OMS/Java: a multi-purpose system
Note that in Fig. 6 the OMS/Java application framework is part of all client applications, and that the storage management component of the OMS/Java server is exchangeable. This means that one can use as a storage management system either an existing database management system such as Objectivity/DB or a persistant Java system such as ObjectStore PSE for Java. In the following sections, we describe those concepts of the OMS/Java system which are important to understand the basic ideas behind the framework as well as the decisions made to facilitate system extensibility. 4.1
Core Architecture
The OMS/Java system is divided into three main components:
OMS/Java : Model Extensibility of OODBMS
125
– The Configuration Component The configuration component consists of all those parts of the system that are exchangeable and extensible, including: • Object Identifier Manager • Algebra Operations • Constraints • Query Language • Data Definition Language • Data Manipulation Language • Data Structures (such as Hashtables) and Data Access Algorithms – The Core System The core system manages all the data structures needed for supporting the OM model. A description of some of these structures is given in Sect. 4.2. Further, special design patterns have been used to connect the core system to the other components (Sect. 4.3). Finally, all features of the system are available through an Application Programming Interface (API). – The Storage Management Component The storage management component is responsible for not only the persistence of all data, but also for Transaction Management, Concurrency Control, Recovery and Security which includes Access Control and Authentication. Since this component is actually designed in such a way that it can be connected to other database management systems, most of these tasks are passed on those systems. 4.2
Basic Concepts
OMS/Java supports the core object model OM (Sect. 3.2) through the notion of OM Objects, Collections and Constraints. OM Objects. An object in the OM model does not represent the same entity as a Java object which is an instance of a Java class. An OM object has a unique object identity during its whole lifetime. This differs from the way object references are handled in Java. A reference to an object in Java is only unique as long as the object is loaded in the Java Virtual Machine. Consider, for example, that you store a Java object in a file and then read it back again from that file. Reading from the file means that the Java Virtual Machine creates a new Java object with a new reference to it so it is not possible to decide whether the object you stored before is the same as the one you retrieved. To solve this problem, we introduced a special class ObjectID. Every instance of this class represents an unique persistent object identifier and is related to exactly one OM object. Further, depending on the context in which an object is accessed (e.g. accessing an object through a specific collection), the object changes its role. This implies that more than one type can be associated with an OM object. This is not supported by the Java environment since it is not possible to change the type definition of a Java class instance during run-time.
126
Andreas Steiner et al.
As is shown in Fig. 7, we solved this problem by splitting up an OM object into the two Java classes OMObject and OMInstance (which is actually a Java interface class). Additionally, meta information about OM objects needed by the OMS/Java system are defined in instances of the Java class OMType. OMObject
1:1
OM Objects
1:1 are_related_to
ObjectID
Object IDs
0:* refer_to 1:1 OMInstance 1:1
OM Instances
0:* are_defined_by
OMType
OM Types
Fig. 7. OM Objects
Another approach would have been to implement one’s own dynamic type system as has been done, for example, in the OMS/Prolog system [NW97]. But, since we wanted to be able to use standard Java classes in our system, this approach was not suitable. Using Java classes involves two steps: first the meta information has to be defined in the schema (see also Sect. 5.2) such as type person ( name: String; );
Then, for all the types defined in the schema, a corresponding Java class has to be specified. This can be done in three ways: – by extending the class OMSimpleInstance which provides all functionality needed by the system such as meta information. – by implementing the interface OMInstance, or – by using the adapter class OMAdapter. An adapter or wrapper “converts the interface of a class into another interface which clients expect. It lets classes work together that could not otherwise because of incompatible interfaces.” [GHJV95]. Hence, the OMAdapter class makes it possible to use most of the Java classes without modifications. The following class definition is an example for the first approach: package demo; import omJava.*; public class Person extends OMSimpleInstance { public String name; }
OMS/Java : Model Extensibility of OODBMS
127
Collections. An OM object can be classified according to different criteria by inserting it into collections. Removing an object from a collection and inserting it into another one simply changes its role. So, if an OM object is stored in a collection then it gains automatically the member type of that collection. Further, a collection itself is an OM object and can therefore also be classified by inserting it into other collections. A rich set of algebra operations over collections (see Sect. 3.2) together with different kinds of collections such as sets, bags and sequences are also provided by the OMS/Java system. Constraints. The various structural constraints such as subcollection, cover, disjoint and intersection between collections (Sect. 3.2), as well as the cardinality constraints for associations, are specified as Global Constraints in OMS/Java and will be checked at commit time of a transaction. Local Constraints, on the other hand, are related to a single OM object. For example, the constraint that all objects in a collection must be of the same type is defined as a Local Constraint and will be checked every time the collection is updated. Local Constraints are in fact specialisations of Triggers. A Trigger is related to one or more OM objects and is activated as soon as the trigger condition holds. Note that Global Constraints, Triggers and Local Constraints are also treated as OM objects in OMS/Java. 4.3
Design Issues
To take full advantage of object-oriented concepts such as reuse and extensibility, it is important to use the appropriate design patterns. “Each pattern describes a problem which occurs over and over again in our Environment, and then describes the core of the solution to that problem, in such a way that you can use this solution a million times over, without ever doing it the same way twice.” [GHJV95] Using design patterns not only increases the quality of the software product, but also makes it possible to build systems that are flexible enough to support requirements not known while implementing it. The following four design patterns were used to achieve extensibility of our system: Abstract Factory. The main construct for system parameterisation, respectively, configuration is the Abstract Factory. It defines a set of abstract parameters needed at run-time by an application or, in our case, the OMS/Java core system. Such parameters include among others the algebra used for operations over collections, the object identifier generator, the query language and data definition language parsers and the various data structures such as hashtables (Sect. 4.4).
128
Andreas Steiner et al.
Abstract Classes. An Abstract Class defines a common interface for its subclasses. It defers some or all of the implementation to the subclasses. Hence, an abstract class cannot be instantiated, but forces the developer of the subclasses to at least implement important methods needed for the overall system functionality. For instance, there is a data structure Container which is used to store the elements of a collection. Inserting elements into such a container depends on whether a collection is a set, bag or sequence. For this purpose the abstract class InsertionStrategy defines a method insert which provides a common interface for inserting objects into containers and which must be specified by its subclasses: abstract public class InsertionStrategy { ... abstract public void insert(Container container, Object object); }
Strategy. The Strategy design pattern defines a family of algorithms. By encapsulating all these algorithms into a strategy object, they become interchangeable and independent from a specific client object. For example, the algorithms to insert objects into collections are defined as insertion strategies making it possible to vary the insertion algorithm depending on whether a collection is a set, a bag or a sequence. public class SetStrategy extends InsertionStrategy { ... public void insert(Container container, Object object) { ... } } public class BagStrategy extends InsertionStrategy { ... public void insert(Container container, Object object) { ... } } ...
Bridge. “A Bridge design pattern decouples an abstraction from its implementation so that the two can vary independently.” [GHJV95] For example, two bridges have been used to separate the algebra operations and the underlying data structures from the collection object. Therefore, it is possible to exchange the data structures and/or the algebra without having to modify the interface of the collection object. This implies that the collection object must be initialised by these two bridges. The following example illustrates the use of these bridge classes. Suppose we want to find all objects of which the attribute name is equal to the value Smith using the algebra operation select: OMCollection result = collection.select("name", "Smith");
OMS/Java : Model Extensibility of OODBMS
129
This invokes the method select of the OMCollection object which calls the corresponding method of the Algebra object. This method then performs the selection and inserts the matching objects into the result collection by invoking the insert method of this collection. That method again redirects the insertion by calling the insert method of the InsertionStrategy object which itself invokes the method insert of the Container object. public class OMCollection extends OMSimpleInstance { protected Algebra algebra; // decouples algebra protected Container container; // decouples data structure protected OMInsertionStrategy strategy; ... public OMCollection select(String selection, Object value) { return algebra.select(this, selection, value); }
}
public OMCollection insert(Object object) { strategy.insert(container, object); }
public class OMAlgebra implements Algebra { ... public OMCollection select(OMCollection collection, String selection, Object value) { OMCollection result = ... // perform select operation over // ’collection’ result.insert(object.key()); // insert matching objects into //result collection return result; } } public class Container { ... public void insert(Object object) {...} }
4.4
The Link between the Core System and the Configuration Component
The Core System is connected to the Configuration Component by an instance of the Java class Factory as shown in Fig. 8. This Factory class corresponds to the design pattern Factory and is an important concept for making a system extensible. The Factory class provides various methods such as getNextObjectID(), getOMLparser(), getCollectionAlgebra() and getCollectionContainer(). The method getNextObjectID(), for instance, returns an instance of class ObjectID which represents a unique object identifier, whereas getCollection
130
Andreas Steiner et al.
Algebra() returns an instance of Algebra which can be used for evaluating algebra operations on collections. All parts of the Configuration Component are obtained by calling the appropriate method of the Factory instance which, for its part, is given as a parameter while initialising the Core System. By subclassing the class Factory and overriding methods such as getNextObjectID(), it becomes possible to provide to the core system, for instance, different data structures such as object identifiers.
Fig. 8. The link between the core system and the configuration component
5
From OMS/Java to TOMS/Java
Extending OMS/Java with new features means in fact to extend, respectively, to replace parts of the Configuration Component which has been described in Sect. 4.1. The following list gives a summary of the main tasks that are involved when implementing extensions to OMS/Java: – Extending the syntax of the Data Definition Language, the Data Manipulation Language and Query Language, and changing the corresponding parsers, i.e. implementing the new features in subclasses of the standard parsers. – Subclassing the standard object identifier class – Implementing new algebra operations in a subclass of the algebra class – Extending the various constraint classes – Defining a subclass of the class Factory which is used as a link between the Core System and the Configuration Component The following sections describe in more detail how OMS/Java has been extended to support the Temporal Object Data Model TOM [SN97c,SN97a]. 5.1
The Temporal Object Data Model TOM
TOM is based on the generic object data model OM [Nor93] and exhibits many of the features found in various temporal object-oriented models, e. g.
OMS/Java : Model Extensibility of OODBMS
131
[RS91,WD93,KS92,BFG96], but in a more generalized form. For example, we timestamp not only data values, but also collections of objects and the associations containing the temporal relationships between objects. Anything considered to be an object in our model may be timestamped, even metadata such as constraints, types or databases. This allows the specification of a lifespan for each instance of such a construct. Hence, TOM is based on object-timestamping, i.e. the object identifiers are extended with a timestamp. A timestamp is a temporal element [Gad86] – a set of time intervals, closed at the lower bound and open at the upper bound. Additionally, TOM supports a temporally complete query lanaguage [BJS95] and temporal constraints. Temporally complete means that all operations found in the non-temporal data model have to be generalized into temporal operations, including, for example, the usually neglected set difference operation. Additionally, temporal comparison predicates have to be supported allowing expressions such as a before b. Temporal constraints generalise their non-temporal counterparts with temporal semantics. A constraint no longer has to hold over a single database state, but rather over the set of those database states contained in the constraint’s lifespan. 5.2
Language Extensions
This section describes the extensions made to the syntax of the various languages (DDL, DML and QL) by means of an example: Suppose we want to build an information system based on the schema given in Fig. 4. For the sake of simplicity, we model only the collections Persons, Employees and Students. Defining the Schema. The following schema has been specified using OML, the data definition language of OMS/JAVA: person = demo.Person; // links type ’person’ to the Java class // ’demo.Person’ employee = demo.Employee; student = demo.Student; type person lifespan {[1980/1/1-inf)} ( name: String; ); type employee subtype of person lifespan {[1980/1/1-inf)} ( empID: Integer; salary: Integer; );
132
Andreas Steiner et al.
type student subtype of person lifespan {[1982/1/1-inf)} ( studID: Integer; ); collection Persons collection Employees collection Students
: set of person lifespan {[1980/1/1-inf)}; : set of employee lifespan {[1980/1/1-inf)}; : set of student lifespan {[1982/1/1-inf)};
constraint: Employees subcollection of Persons lifespan {[1980/1/1-inf)}; constraint: Students subcollection of Persons lifespan {[1982/1/1-inf)};
Three categories of objects are defined in this schema: type, collection and constraint objects. Further, the Lifespan of TOM objects, specifying the period of time in which an object is valid, is set by the corresponding lifespan parameter. For supporting the notion of Lifespans, we first had to extend the syntax of OML by the following definitions given in EBNF [Wir96]: lifespan = "{" interval {"," interval} "}". interval = "[" date "-" date ")". date = (year "/" month "/" day ["˜" hour ":" minute ":" second]) | "inf".
Next, the OML parser had to be adapted to the new syntax. Since all parsers in our system are part of the Configuartion Component and not of the Core System, no changes are necessary in the Core System classes for supporting, for example, a new syntax. Only the corresponding parser has to be changed which can be achieved by either replacing the existing parser or by extending it. In most cases, especially if there are only minor changes of the syntax, extending an existing parser will be possible because we designed the standard parsers in such a way that they invoke special methods during the syntax analysis. For example, after the OML parser has parsed a type, collection or constraint definition such as type person, the method parseRestOfLine(...) is called. In the case of the standard parser, this method just returns without performing any action. Hence, for supporting the new Lifespan syntax we just extended the Java class OMLparser and overrode the parseRestOfLine(...) method. Creating Objects. One possibility for creating objects and inserting them into collections is to import a file in which the data is described in terms of Data Manipulation Language statements: create object paul lifespan {[1969/1/1-inf)} dress object paul as demo.Person during {[1969/1/1-1975/6/1)} values (name = "Paul Jones")
OMS/Java : Model Extensibility of OODBMS
133
dress object paul as demo.Employee during {[1986-1996)} values (empID = 23; salary = 2000) dress object paul as demo.Student during {[1993-1995)} values (studID = 2674) insert object paul into collection Persons during {[1985/1/1-1996/1/1)}
The create object statement defines an OM Object the alias of which is paul. Its lifespan is set to the Lifespan parameter lifespan [1969/1/1-inf). The dress object statements further specify that instances of the Java classes demo. Person, demo.Employee and demo.Student are associated to that object and that this association is valid as given by the During parameters. Remember that an OM Object can refer to more than one instance of a Java class (as shown in Fig. 7). The insert statement states that the object paul is a member of the collection Persons during the time period specified by the During parameter during [1985/1/1-1996/1/1). Note that the collection Persons has been already created after loading the schema. To support the Lifespan as well as the During parameters, which are features of TOM, we had to extend the DML Parser taking the same approach as in the case of the OML Parser, i.e. we extended the DML Parser by making a subclass of the class DMLparser and implemented, respectively, overrode some methods. Querying the Database. AQL (Algebraic Query Language) is the query language of OMS/Java. This language was originally developed for OMS/Prolog [NW97], and it is both a simple, and yet powerful language with operations over values, objects, collections and associations. As a simple example, consider that we want to know which objects are classified as both Employees and Students. This can be expressed in AQL as: Employees union Students;
In the case of TOM the corresponding query contains the keyword valid in addition: valid Employees union Students;
The valid keyword denotes that the query should be evaluated only on those objects which are valid now. The result of this query is presented by TOMS/Java in the form of a scrollable list. By double-clicking on one of the items in the list, the system dynamically generates a window showing all properties of the selected object. Figure 9 gives an example of such an object. As in the case of DDL and DML, we have extended the Java class AQLparser, in this case to support temporal queries.
134
Andreas Steiner et al.
Fig. 9. A person object
5.3
Changes Made in the Configuration Component
In this section, we outline the changes made in the Configuration Component that have been necessary to support TOM. Table 2 gives an overview of the classes which had to be extended. Instances of these classes can be obtained by calling the corresponding method of the Factory. Note that the term Factory is used in the text as a placeholder for an instance of the class ValidTimeFactory. This class is a subclass of the OMS/Java class Factory and overrides methods such as getNextObjectID() to support the semantic of TOM (Sect. 4.3). Table 2. Java classes which had to be extended
OMS/Java class OMLparser DMLparser AQLparser ObjectID InstanceList Algebra SubcollConstr
TOMS/Java class TOMLparser TDMLparser TAQLparser ValidTimeObjectID ValidTimeInstanceList ValidTimeAlgebra ValidTimeSubcollConstr
Object Identifiers. When a schema definition file is loaded, the core system first invokes the method getOMLparser() of the Factory to retrieve an instance of an OML parser, followed by a call to the parser method generate(schema) which parses the schema and generates the objects. In the case of TOM, this parser must be able to manage temporal information such as Lifespans. Hence, we made a subclass of OMLparser containing new methods for temporal information processing as described in Sect. 5.2. For the schema given in Sect. 5.2, the parser creates the following Java instances: instances of OMType for each type definition, instances of OMCollection for each collection and instances of OMSubcollConstr for each subcollection constraint. For each of these instances, the parser also gets an ObjectID instance, respectively in the case of TOM, a ValidTimeObjectID instance by calling the Factory method getNextObjectID(). The class ValidTimeObjectID is a subclass of ObjectID which has been extended for storing Lifespan information. Finally, the parser sets the lifespans of these object identifiers to the ones given in the schema.
OMS/Java : Model Extensibility of OODBMS
135
Object Creation. One way of creating objects is to import a file containing data defined by Data Manipulation Language statements as has been described in Sect. 5.2. For this purpose, the core system calls the Factory method getDMLparser() which returns an instance of the class DMLparser. Actually for TOM, the Factory returns a subclass of DMLparser which provides all methods needed for managing temporal information such as Lifespans (Sect.5.2). The method generate(inputFile) of the parser is invoked by the core system which imports, respectively, creates the objects. For our example, the parser first creates an OMObject instance, the alias of which is paul, and then associates it to an instance of ValidTimeObjectID of which the lifespan is set to [1969/1/1-inf). Further, the parser creates instances of the Java classes demo.Person, demo.Employee and demo.Student and sets the values of the different instance fields by calling the setAttrib(...) method of each instance. Finally, the parser relates these instances to the object paul by calling the method dress(instance) which inserts an instance into the Instance List of the object paul. The data structure Instance List, which is obtained by the Factory method getInstanceList() while creating an OM Object, is used to manage all OM Instances related to this object and provides methods such as insert(OMInstance instance), delete(OMInstance instance), update(OMInstance instance) and replace(OMInstance instance, int index). For supporting TOM, we had to extend the StdInstanceList class and to override these methods. In our example, the Validity of the relationship between the object paul and the associated instances of demo.Person, demo.Employee and demo.Student is specified by the During parameters. These parameters are stored by the parser in the corresponding instance by calling the method setParameter(parameter) of the OMInstance class. Note that this parameter is of type Object making it possible to store any kind of information in an OM Instance object. Collections and Algebra Operations. Collections are represented by instances of the Java class OMCollection and are created, for example, while loading a schema definition file by the OML parser. Creating such an object invokes the constructor of the OMCollection class. This constructor itself calls the Factory methods getCollectionContainer() which returns an instance of the class Container needed for managing the members of a collection, and getCollectionAlgebra() which returns, in the case of TOM, an instance of the class ValidTimeAlgebra providing algebra operations such as union(...). Since the data structure as well as the algebra are returned by the Factory and are therefore not part of the OMCollection class which belongs to the core system, it is possible to exchange, for instance, the algebra without having to change the OMCollection class. Inserting objects into collections can be done, for example, by importing a data file using the DML parser. In our example, the object paul is inserted into the collection Persons. In addition, the during parameter, which is a feature of TOM, specifies the time period during which paul is a member of Persons. No changes were necessary to support this feature because an OMCollection
136
Andreas Steiner et al.
class provides a method setParameter(object, parameter) which can be used to store additional information for each member object such as the Validity given by the during parameter. This information is needed by the algebra class and can be retrieved by calling the collection method getParameter(Object object). Hence, the OMCollection class does not need to be able to manage this information and since the parameter is of type Object, the stored information is not restricted to specific semantics such as temporal information. So, for inserting the object paul into Persons the DML parser first invokes the method insert(object) of the collection to insert the object and then sets the Validity given by the during parameter by calling the collection method setParameter(object, parameter). Evaluating algebra operations simply means calling the appropriate methods of collection objects. For example, the AQL parser evaluates the temporal AQL query “valid Employees union Students;” by invoking the union(...) method of the Employees collection object: employees.union(students, algebraParameter);
The method union takes two input parameters: the second collection object for the union operation and an algebra parameter which is in our case the keyword valid. Further, this method calls the method union(employees, students, algebraParameter) of the algebra instance which has been specified when the collection object has been created. Hence, only class Algebra, or subclasses of it such as ValidTimeAlgebra, must know how to handle the algebra parameter. Note that this parameter can also be of any type. Constraints. Global Constraints, Triggers and Local Constraints, which have been defined in Sect. 4.2, are treated as OM Objects and are therefore associated to an unique object identifier which in the case of TOM is an instance of ValidTimeObjectID. Global Constraints such as the subcollection constraint Employees subcollection of Persons lifespan [1980/1/1-inf) in our example are created by the OML parser after loading the schema definition file. The parser invokes for this purpose the Factory method getSubcollConstr() which returns, in the case of TOM, an instance of OMValidTimeSubcollConstr. The Lifespan parameters denote the time periods in which the constraints are valid. The OMValidTimeSubcollConstr class is a subclass of OMSubcollConstr of which various methods have been overridden for supporting temporal semantics. Triggers and Local Constraints are created at the same time when the objects to which they belong are initialised.
6
Conclusions
We presented a general approach for extending database management systems (DBMS) in terms of model extensibility and described the corresponding general
OMS/Java : Model Extensibility of OODBMS
137
DBMS architecture. While our approach builds on ideas of architecture extensibility, i.e. providing architectural support in terms of services such as storage management, it also differs from it in terms of focussing on the generalisation of a core data model and system for advanced application domains. In particular, we described the OMS/Java system which has been developed to support model extensibilty. As a proof of concept, we have extended OMS/Java to support the temporal object data model TOM. We are convinced that using the model extensibility approach leads to database management systems which are easily adaptable to various application domains. In the future, we want to investigate extending the system for a spatial data model and also for a version model. The latter is intended to provide an OMS/Java implementation platform for a document management system currently under development.
References [BFG96]
E. Bertino, E. Ferrari, and G. Guerrini. A Formal Temporal Object-Oriented Data Model. In P. Apers, M. Bouzeghoub, and G. Gardarin, editors, Advances in Database Technology, pages 342–356. Springer, 1996. [BJS95] M. B¨ ohlen, C. Jensen, and R. Snodgrass. Evaluating the completeness of TSQL2. In J. Clifford and A. Tuzhilin, editors, Recent Advances in Temporal Databases, pages 153–172, 1995. [BLW88] D.S. Batory, T.Y. Leung, and T.E. Wise. Implementation Concepts for an Extensible Data Model and Data Language. ACM TODS, 13:231–262, September 1988. [CDG+ 89] M. J. Carey, D. J. DeWitt, G. Graefe, D. M. Haight, J. E. Richardson, D. T. Schuh, E. J. Shekita, and S. Vandenburg. The EXODUS Extensible DBMS Project: An Overview. In S. Zdonik and D. Maier, editors, Readings in Object-Oriented Database Systems. Morgan-Kaufmann, 1989. [CDKK85] H.-T. Chou, D.J. DeWitt, R.H. Katz, and A.C. Klug. Design and Implementation of the Wisconsin Storage System. Software Practice and Experience, 1985. [Cha96] A. Chatterjee. A Framework for Object Matching in Federated Databases and its Implementation. Journal of Intelligent and Cooperative Information Systems, 1996. [EN94] R. Elmasri and S. Navathe. Fundamentals of Database Systems. Benjamin/Cummings Publishing Company, 1994. [Gad86] S. K. Gadia. Toward a Multihomogeneous Model for a Temporal Database. In Proceedings of the International Conference on Data Engineering, pages 390–397, 1986. [GD94] A. Geppert and K. Dittrich. Constructing the Next 100 Database Management Systems: Like the Handyman or Like the Engineer. ACM SIGMOD Record, 1994. [GHJV95] E. Gamma, R. Helm, R. Johnson, and J. Vlissides. Design Patterns, Elements of Reusable Object-Oriented Software. Addison-Wesley, 1995. [HCL+ 90] L.M. Haas, W. Chang, G.H. Lohman, J. McPherson, P.F. Wilms, G. Lapis, B. Lindsay, H. Pirahesh, M.J. Carey, and E. Shekita. Starburst Mid-Flight: As the Dust Clears. IEEE Trans. on Knowledge and Data Engineering, 1990.
138
Andreas Steiner et al.
[HS95] [KS92] [MBH+ ]
[Mes97] [Nor93] [Nor95]
[NW97] [Pro97] [RS91] [SC94]
[SN97a] [SN97b] [SN97c] [WD93]
[Wir96]
A. Heuer and G. Saake. Datenbanken: Konzepte und Sprachen. International Thomson Publishing, 1995. W. K¨ afer and H. Sch¨ oning. Realizing a Temporal Complex-Object Data Model. In SIGMOD Conference 1992, pages 266–275, 1992. F. Maryanski, J. Bedell, S. Hoelscher, S. Hong, L. McDonald, J. Peckham, and D. Stock. The data model compiler: A tool for generating objectoirneted database systems. In Proc. of the 1986 Intl. Workshop on ObjectOriented Database Systems. S. Messmer. The OM Model as a Framework for the Java Environment. Master’s thesis, Department of Computer Science, ETH Zurich, 1997. M. C. Norrie. An Extended Entity-Relationship Approach to Data Management in Object-Oriented Systems. In Proceedings of the 12th International Conference on the ER Approach, 1993. M. C. Norrie. Distinguishing Typing and Classification in Object Data Models. In Information Modelling and Knowledge Bases, volume VI, chapter 25. IOS, 1995. (originally appeared in Proc. European-Japanese Seminar on Information and Knowledge Modelling, Stockholm, Sweden, June 1994). M. C. Norrie and A. W¨ urgler. OMS Object-Oriented Data Management System. Technical report, Institute for Information Systems, ETH Zurich, CH-8092 Zurich, Switzerland, 1997. R. Prodan. OMS/Objectivity. Master’s thesis, Department of Computer Science, ETH Zurich, 1997. E. Rose and A. Segev. TOODM - A Temporal Object-Oriented Data Model with Temporal Constraints. In Proceedings of the 10th International Conference on the ER Approach, 1991. A. Segev and A. Chatterjee. Supporting Statistical Operations in Extensible Databases: A Case Study. In Proc. IEEE Seventh International Working Conference on Scientific and Statistical Database Management, Charlottesville, USA, sep 1994. A. Steiner and M. C. Norrie. A Temporal Extension to a Generic Object Data Model. Technical Report 265, Institute for Information Systems, ETH Z¨ urich, May 1997. A. Steiner and M. C. Norrie. Implementing Temporal Databases in ObjectOriented Systems. In Database Systems for Advanced Applications (DASFAA), 1997. A. Steiner and M. C. Norrie. Temporal Object Role Modelling. In Proceedings of the Conference on Advanced Information Systems Engineering (CAiSE), 1997. G.T.J. Wuu and U. Dayal. A Uniform Model for Temporal and Versioned Object-Oriented Databases. In A. Tansel, J. Clifford, S. Gadia, S. Jajodia, A. Segev, and R. Snodgrass, editors, Temporal Databases: Theory, Design, and Implementation, chapter 10, pages 230–247. Benjamin/Cummings Publishing Company, 1993. N. Wirth. Compiler Construction. Addison-Wesley, 1 edition, 1996.
An environment for designing exceptions in work ows ? F. Casati, M.G. Fugini, I. Mirbel
fcasati,fugini,[email protected]
Politecnico di Milano Piazza L. Da Vinci, 32 - I-20133 Milano, Italy
Abstract. When designing a WorkFlow schema, often exceptional sit-
uations, such as abnormal termination or suspension of task execution, have to be dealt with by the designer explicitly, as well as operations which typically occur at the beginning and termination of tasks. This paper shows how the designer can be supported by tools allowing him to capture exceptional behavior within a WF schema by reusing an available set of pre-con gured exceptions skeletons. Exceptions are expressed by means of triggers, to be executed on the top of an active database environment; triggers can autonomously treat exceptional events or typical start/termination situations arising in a WF. In particular, the paper deals with the treatment of typical WF exceptional situations which are modeled as general exception skeletons to be included in a new WF schema by simply specializing or instantiating them. Such skeletons, called patterns, are stored in a catalog; the paper describes the catalog structure and its management tools constituting an integrated environment for pattern-based exception design and reuse.
1 Introduction WorkFlow (WF) systems are currently becoming a more and more applied technology in the direction of providing a common framework for supporting activities within oce information system. WFs in the organizations are modeled to capture the possible sequences of activities needed to reach a given goal, and to provide tools and links to the existing Information Systems of the organization. Many aspects have to be considered in conjunction with the pure activity ow. In particular, the documents and information associated with the activities, the performers of the activities and their role in the organization, and the possible exceptions to the normal ow of activities. In other related elds, such as in Software Engineering and Information Systems, analogous techniques and tools are used combining data and function modeling to represent the application functionalities and to represent both normal and abnormal situations and exceptions[14]. Currently, few techniques and tools exist for WF design. Some of them [17] are based on rules and catalogs of prede ned WF portions for WF management. ?
Research presented in this paper is sponsored by the WIDE Esprit Project N. 20280
B. Pernici and C. Thanos (Eds.): CAiSE’98, LNCS 1413, pp. 139-158, 1998. Springer-Verlag Berlin Heidelberg 1998
140
F. Casati, M.G. Fugini, I. Mirbel
In general, the designer of a WF schema has to provide many details in order to obtain a schema which is executable by a WorkFlow Management System (WFMS). This description requires the speci cation of both the normal ow and of the possible variations due to exceptional situations that can be anticipated and monitored. This paper addresses the issue of providing support in designing exceptions in a WF focusing on exceptions which can be statically foreseen. It describes an environment of tools for the speci cation of WF schemas. Exceptions can be included in a schema by reusing (i.e., selecting and customizing) a set of pre-de ned generic exceptions. The term \exception" is used in a broader sense than simply for exceptional situations. In fact, exceptions model also conditions regarding the starting, termination, or repetitions of WF instances (called cases) or regarding the periodical occurrence of operations in the WF execution. The support environment, called , consists of a catalog of pre-designed WF schema portions, and of tools for accessing the catalog. We are mainly concerned with the speci cation of exceptional situations in WFs, that is, situations where events occuring in a WF schema may cause a task to be canceled, or a WF case to be terminated. Such exceptional situations may be designed in advance and inserted in the WF schema speci cation by including exception handlers for the most likely occurring events or by designing typical termination handling strategies. Another situation is the WF start phase which can be subject to typical situations. For instance, a WF has to be started periodically, or at a given time. These situations may be included as a pre-set part of the schema speci cation. The catalog of the environment stores WF schema portions, mainly dealing with exceptions, to be reused during the design of a new schema. The contents of the catalog is a set of typical patterns[16] enabling the designer to deal with frequent occurrences of similar situations in WF design in given application domains, and with generic situations that may occur in any application domain. The catalog, and its related design support tools, is based on the following main issues:
werde
{ rules: rules model exceptions, starting and termination situations, and WF
schema evolution due, for instance, to changed requirements deriving from new types of exceptional events; { patterns: typical rules may be used in several designs. Patterns described in the paper are rules, or sets of related rules, to control a given typical situation. Patterns are a way for the designer to specify the WF schema by reusing previous design experience. Rules in WF modeling have been considered, among others, by [12]. However, they are mainly used to represent the complete ow. In other approaches to WF modeling ([13, 15]), a strong emphasis is put on providing a simple and graphic interface to WF design. However, if such interfaces are to capture all the details of anomalous ows, readability is penalized, such as in Petri Net based approaches ([2]).
An Environment for Designing Exceptions in Workflows
141
The approach of this paper based on pattern reuse aims at improving the speed and quality of WF schema design. A pattern is a generalized description of a rule or sets of rules that can be associated to a WF schema, either to model exceptions, or to model starting and termination situations, or situations to be controlled due to schema evolution. Reuse, also based on patterns, is beeing successfully applied to software development ([5, 18, 19]). The paper focuses on the functionalities of the environment which allow the designer to reuse patterns stored in a catalog for dealing with exceptions in WF schema design. Functionalities for pattern design are also presented. The environment is presented mainly in terms of speci cations of its functionalities; then, the implemented functionalities are presented in detail. The paper is organized as follows. In Section 2, we illustrate the catalog of patterns, and describe the use of rules in WF design by presenting exceptions and patterns. In Section 3, we present the functionalities of the environment for access and management of the catalog and show the tools usage for exception design in WF schemas and for exception analysis. Finally, in Section 4, we comment and give some hints about future work.
werde
werde
2 A Pattern Catalog
werde
This section describes the pattern catalog of the environment where a set of pre-de ned WF patterns is stored at di erent levels of abstraction and genericity in a layered structure. The catalog is the centralized repository of reusable WF elements in order to support speci cation by example of di erent elements of a WF, at di erent granularity levels, such as process variables, exceptions, and tasks. Exceptions and patterns will be described in the following of this section. Patterns model situations that can occur in WF design in an applicationindependent way, at di erent levels of genericity (e.g., integrity constraints holding for schemas of any application domain, or schema fragments describing frequently occurring combinations of tasks in a given application domain). They can be described either using a language for specifying exceptions and/or through the WF process description language WPDL [6]. In Figure 1 the categories of patterns stored in the catalog are depicted. Patterns are classi ed at three di erent layers.
{ Built-in patterns. Patterns at this level deal with reaction to events and time
alarms (Basic patterns), integrity and security constraints over WF data (Authorization, Integrity, and Count patterns), WF management (Starting, Interaction, and Continuation patterns), and with changes in the organization structure (Organization and Schema evolution patterns). They describe context-dependent and time-dependent knowledge related to basic exceptional situations independent of the semantics of all possible applications. Built-in patterns specify, for example, basic interactions between WF elements, or situations where a task has to be suspended or resumed, or express
Document preparation Document revision Reservation Other patterns
Fig. 1. The werde pattern catalog exception requirements related to WF interoperability. Built-in patterns are implemented as generic rules that can be adapted to di erent application contexts. { Generic tasks. Patterns at the generic tasks layer allow the description of frequently occurring activities in WF design in the form of tasks, independent of the documents involved in their execution (e.g., a task for registering information in a document). { Generic application-oriented structures. This layer of the catalog contains WF fragments and associated exceptions needed to model frequently occurring situations of one or more application domains. These are frequently occurring combinations of tasks (i.e., WF fragments) generally performed together to achieve an organization goal in a given application domain (e.g., a generic structure describing the concurrent review of a document by several people). In order to be accessible and user friendly for the designer, the catalog contains both a Pattern Speci cation element and a Sample Usage element. The rst one describes the exception and includes parts allowing the designer to understand the goal of the pattern, to locate the pattern in the catalog, and to link patterns together within a WF schema. The second element contains sample instantiations of patterns in various application domains, thus guiding the designer in completing a new schema.
An Environment for Designing Exceptions in Workflows
143
In the following of the paper, we describe exceptions and patterns. In particular, we show how exceptions are speci ed and used, and how exceptions are then abstracted in patterns. For details about the patterns contained in the various categories of the catalog, the interested reader can refer to [9].
2.1 Exceptions
A short introduction to the exception language is now given (see [7] for a detailed description). The Chimera-Exc exception language is based on the Chimera language for active object-oriented databases. An exception is speci ed by an eventcondition-action (ECA) rule (also called trigger in the following). Events denote the occurrence of a possibly exceptional situation; conditions determine if the occurred event actually corresponds to an exceptional situation to be managed, while the action part de nes the operations to be performed in order to manage the exception. ECA rules may refer in their event, condition, or action part, to objects and classes of the WFMS database. These include static and dynamic information on WFs and tasks (stored in objects of classes case and task respectively), data relating WF participants (objects of class agent), plus a WFspeci c class, bearing the name of the WF, whose objects store process-speci c variables local to each single case. For instance, variables needed for the execution of cases of a hotel roomReservation WF, such as the room number or the customer name, are stored as attributes of an object of the roomReservation class. An exception can be sensitive to di erent kinds of events: data events are raised upon modi cation of the content of the WFMS database. Data events include the creation of a new object (create), the modi cation of an object's attribute (modify), and the deletion of an object (delete). For instance, event modify(task) is raised when an attribute of an object of the task class is modi ed. Work ow events are raised when a case or task is started or completed. WF events include caseStart, caseEnd, taskStart, and taskEnd. For instance, event taskStart(myTask) is raised when an instance of task myTask is started. Temporal events are raised upon the occurrence of a de ned temporal instant. They can be expressed as deadlines, temporal periods, or interval elapsed since a speci c date or time. For instance, event elapsed 1 day since taskStart(myTask) is raised as one day has elapsed since the start of an instance of task myTask. Finally, external events, quali ed by their name, are noti ed by external applications: these may correspond to applicative situations, such as a phone call by a customer cancelling a reserved room. An example of external event is raise(cancelReservation). The condition part consists of a declarative query, expressed by a formula evaluated over the WFMS database state. The formula is expressed as a conjunction of predicates, and includes a set of variables to which bindings are associated as a result of the formula evaluation: if the result of the query is empty (i.e., if no bindings are produced), then the condition is not satis ed, and the action part is not executed. Bindings resulting from the formula evaluation are passed to the action part in order to perform the reaction over the appropriate objects.
144
F. Casati, M.G. Fugini, I. Mirbel
The condition includes class formulas, for declaring variables ranging over the objects of a speci c class (e.g., task(T)), type formulas for declaring variables of a given type (e.g., real(R)), and formulas expressing comparisons between expressions (e.g., T.executor="Lisa"). Objects a ected by events are captured in the condition part by the occurred predicate. Thus, for instance, in an exception triggered by the caseStart event, the binding with the started instance is obtained by means of the predicates case(C), occurred(caseStart,C). The action part includes data manipulation actions, updating the content of the WFMS database, and operations executed through calls to the WF engine. The rst category includes the primitives create, delete, and modify. For instance, delete(Log,L) deletes all objects of class Log to which L is bound after the condition evaluation. The second category includes primitives to start, rollback, or suspend the execution of cases and tasks, assign or delegate the execution of tasks, and send messages to agents. Examples are delegateTask(T,A), startCase(mySchema, startingParameter), and notify(T.executor, "deadline is approaching"). A sample trigger activated at the end of cases of the roomReservation WF is as follows. The condition determines which is the case that has caused the triggering (predicates case(C), occurred(caseEnd,C)), retrieves the object storing the variables of the just completed case (roomReservation(R), R.caseId=C) and then checks if the customer has not paid (R.paymentDone=FALSE). If the condition is veri ed (i.e., if bindings are produced), then the action part is executed, causing the start of an instance of the cancelBooking WF and providing the customerID as input parameter. define trigger missingPayment events caseEnd condition case(C), occurred(caseEnd,C), roomReservation(R), R.caseId=C, R.paymentDone=FALSE actions startCase("cancelBooking",R.customerId) end
Rules can be associated to a task, to a schema, or to the whole WFMS. Tasklevel rules model exceptions which are related with a single task; an example is a rule reacting to the task deadline expiration. WF-level rules de nes a behavior common to every task in a schema, or model exceptions related to the whole schema; for instance, a WF-level rule could specify that all tasks of a the WF should be suspended at 7pm. Finally, global (WFMS-level) rules model behaviors de ned for every task or case, or exceptions involving cases of di erent WFs. For instance, an exception stating that \tasks that were assigned to an agent who left or is on vacation must be reassigned" should be declared at the WFMS-level, since it de nes a general policy valid for every task. Rules can be ordered by absolute priority. Upon concurrent trigger, the scheduler executes rst the rules with higher priority. Among rules with the same priority, the choice is non-deterministic.
An Environment for Designing Exceptions in Workflows
145
2.2 Patterns Since the Chimera-Exc language is quite exible, the de nition of a variety of exceptional behaviors is allowed. However, experience in exceptions design has shown that, although all the language features are required, most triggers follow common, repetitive \patterns". These observations led to the idea of taking advantage of repetitive patterns in triggers de nition, in order to reduce the design e ort and to provide guidelines for pattern reuse. We introduce patterns as generalized descriptions of triggers and exceptional situations that can frequently arise in WF modeling. Patterns prede ne typical rules or set of rules that capture the knowledge about the occurrence of an exceptional situation and the actions that can be performed to deal with it. Patterns consist of prede ned parts, parameterized parts, and optional parts. Parameterization and optionality are introduced to support pattern re-usability and adaptation in WF design for the aspects related to exception modeling and handling.
Pattern Description: The pattern are mainly described by a speci cation part and a sample usage as follows.
Speci cation part: The speci cation part of the pattern is the description of an exception or of an application, including a set of textual elds and a WPDL and/or Chimera-Exc portion implementing the pattern. In particular, the pattern speci cation is structured in the following elds: { name, uniquely identifying the pattern in the catalog of all patterns; { intent, a textual part describing the purpose of the pattern (intended scope and known uses); { classi cation, according to the categories and sub-categories of the catalog; { template, containing the core speci cation of the pattern. It can be provided in two forms: For patterns representing exceptions, the template contains the speci cation in terms of events, conditions, and actions. On the contrary of the events and conditions, which are the main parts of the patterns, the action part provides only suggestions. This re ect the fact than exception patterns focus on how to capture exceptions, more than on how to x reactions, which are application dependent: in a given application, a warning message can be a satisfying reaction, when a cancelation of the case can be more suitable in another application of the same pattern. For patterns representing application structures, the template contains the diagrammatic WF schema skeleton describing the application fragment. The template usually contains parametric elds to be lled in with speci c values provided by the user; mandatory and optional parts can also be speci ed.
146
F. Casati, M.G. Fugini, I. Mirbel
{ keywords, which are a set of user-selected terms that can be used to refer
(select, search, etc.) to the available patterns in the catalog; this eld allows one to describe more precisely the topics of the pattern, especially to distinguish the di erent patterns of a given category in the classi cation; { related to, establishing links among patterns to combine them in di erent structures; { guidelines, providing suggestions to the user about possible usage and personalization of patterns. In Figure 2, an example of pattern speci cation is given. It is a pattern to be used to start a new case upon termination of another one. Pattern Specification Name: Termination Intent: This pattern allows the definition of the initiation of a case upon termination of another case, considering its status (in terms of WF variables). Classification: Continuation pattern | Termination Template: define trigger termination events caseEnd conditions case(C), occurred(caseEnd,C), <wfName>(W), W.caseId=C, W.= startCase(,wfName>, <parameters>) actions end Keywords: case termination Related to: Guideline: The condition may contain clauses related to values of one or more variables in the terminated case.
Fig. 2. The Termination pattern The complete description of the pattern-speci cation language can be found in [8]. Sample usage part: Since the user of the catalog is intended to be an expert of the application under design and is not required to have a detailed knowledge of the Chimera-Exc syntax, the pattern model is endowed with the user oriented sample usage pattern. This is a set of instantiations of patterns on speci c examples. They show how patterns can be personalized in di erent contexts and applications by illustrating how parameters of patterns can be supplied by the designer to produce a concrete WF. The sample usage part of the pattern description is a set of WF-speci c instantiations of patterns related to an application domain. In general, several examples are provided for a pattern for di erent domains. The sample usage is an instantiation of the variables/parameters appearing in
An Environment for Designing Exceptions in Workflows
147
the template eld of the pattern speci cation according to the domain selected to construct the example. The missingPayment trigger given in Sec. 2.1 is a sample usage of the pattern termination presented in Figure 2.
Pattern Reuse: Two basic mechanisms are provided for reusing patterns in WF
design: specialization and instantiation. Pattern specialization is a mechanism for creating a new pattern starting from an existing one. Pattern instantiation is a mechanism for creating triggers for a particular situation by using a pattern. Pattern specialization: When designing triggers, one may want to specialize a trigger previously de ned. Since it was already explained, the action part of the trigger provides only suggestions, on the contrary of the events and conditions parts. Therefore, the specialization focuses on the event and condition parts. Three ways are given to specialize a trigger. They can be combined. { The rst way consists in rewriting the trigger using more speci c genericterms. Consider a pattern composed of a trigger with an event part containing the generic term . If one wants to write another pattern, with the same structure as the previous one, but focused on external events only, one can specialize the exception part of the pattern by replacing with the more specialized generic-term <externalEvent>. { The second way consists in instantiating part of the trigger. This is done by replacing a generic term by a non-generic one. For example, in a given trigger of a pattern can be specialized into task in a more speci c pattern dealing only with tasks. { The third way consists in adding conditions and actions to a given trigger in order to obtain a more specialized one. These additions can be done only in the condition and actions parts. Adding events is not allowed since this would change the invocation of the trigger. Pattern instantiation: Patterns are used when designing a new WF as follows. After choosing the suitable pattern, this has to be instantiated. In general, this is done by replacing the generic-term with a value, for example becomes documentArrival. A more speci c name must also be assigned to the pattern and the triggers. If there are optional parts, the user has to decide if he/she takes the optional parts or not. If several possible actions are de ned in the action part, it is also necessary to choose one or several of these actions. It is also allowed to add conditions or actions to the trigger, for example to test the value of a variable which is speci c to the WF under design. A more complete and formal description of the instantiation and specialization mechanisms can be found in [8].
3 Speci cation of the Exception Design Environment In this section, we present the environment supporting the pattern-based approach illustrated in Section 2. The tool is mainly given in terms of speci cation of its functionalities which deal with pattern management in the catalog
148
F. Casati, M.G. Fugini, I. Mirbel
and with the design and analysis of exceptions. The current prototype environment implementing most of the functionalities for catalog management, and for pattern-based exception design and analysis is nally described.
3.1 Management of Catalog Management of the pattern catalog deals mainly with the insertion, modi cation and deletion of patterns. However, it also takes into account the management of the categories useful to classify and retrieve patterns. In this section, the environment functionalities dealing with management of categories are presented, followed by the functionalities for pattern management. Category management: As presented in Section 2, the catalog is divided in categories (built-in patterns, generic tasks and generic-application oriented structures). Each is divided in sub-categories: Termination, for example is a sub-category of the built-in patterns category. Since the catalog evolves during time, it is possible to add new categories and sub-categories, and to remove some of them. For removal, a check is performed to ensure that no pattern of the catalog refers to the category or sub-category being deleted. Pattern management: This functionality deals with creation, modi cation and deletion of patterns. When creating a pattern, its di erent parts have to been ll-in by the user: name, intent, classi cation, template, keywords, related-to patterns and guidelines. The pattern name must be unique. The classi cation conforms to the categories and sub-categories of the catalog. Patterns keywords can be selected from a tool-proposed list of keywords and belong to a thesaurus automatically updated whenever new keywords are speci ed for newly de ned patterns. Guidelines can be textual information to be compiled o -line by the pattern designer. Links between templates (\related-to" elds) can be de ned by activating an \add link" and starting a corresponding dialog window. Sample usages can also be written to facilitate pattern reuse. If the specialization mechanism is used to create a new pattern, an existing pattern P can be selected, and the designer can select the parts of P that must be modi ed. The pattern has to share at least one keyword of its generic pattern (the pattern from which it is derived) and has to be placed in the same category/sub-category. Links are maintained in the catalog between patterns belonging to a specialization chain. When modifying patterns, in addition to the modi cation of its features (intent, template, keywords, related-to elds and guidelines), new sample usages can be added to an existing template, by de ning an \executable example" and by selecting the template of interest from the catalog. It is also possible to change the category and/or sub-category of a pattern. This means changing the location of the pattern in the repository, but also changing the location of the patterns linked to it through specialization or generalization, in order to maintain the catalog consistency.
An Environment for Designing Exceptions in Workflows
149
Deletion of patterns is performed by selecting the pattern of interest. The tool then performs a set of basic integrity checks: { if another pattern uses the current one, the link is also removed; { if the pattern is involved in an inheritance hierarchy (is inherited by at least a pattern p2 , and possibly inherits also from at least pattern p1 ), the user has to choose (i) to delete also p2 , or (ii) to build a new inheritance link between p1 and p2 .
3.2 Design of Exception The environment support the pattern-based approach to the design of exceptions in WF schemas limited to the design of its exceptions. In the following, the word exception schema will represent the set of exceptions related to a given WF schema. Exceptions can be designed following a bottom-up process, based as much as possible on reusing the patterns available in the catalog through the instantiation mechanism. The tool guides the designer during the process of exception speci cation by providing both classi cation information, which facilitates the search in the catalog, and guidelines to enter values for pattern parameters and to compose patterns into more complex structures. When creating an exception schema, exceptions can be written (created) from retrieved patterns. These patterns are instantiated/personalized for the WF schema at hand. New exceptions can also be de ned from scratch, if the available patterns do not t the WF requirements. Two search modalities are provided, namely, keyword-based and classi cationbased search. The rst relies on a pre-de ned thesaurus of terms loaded at pattern speci cation time to characterize the pattern within the catalog. Keywords can refer either to pattern elements (e.g., names of events, actions) or can be more generic, domain-related keywords. The second modality allows for the retrieval of patterns based on two orthogonal views of the catalog: { typology of pattern, with respect to event types (e.g., temporal, external, data); { abstraction level of patterns in the catalog (built-in, generic, applicationoriented). Patterns matching search criteria are provided and can be manipulated by the designer. The instantiation consists in binding the pattern parameters on the basis of the semantics of the WF schema, by exploiting the syntax-driven capabilities of the tool and its interface facilities. A set of dialog windows drives the designer in entering the patterns elds, possibly enabling the visualization of the guidelines and usage examples associated with the template under manipulation. New conditions and actions can be added to the exceptions derived from the patterns. The tool dialog facilities support also the de nition of new exceptions from scratch. Triggers within a given exception schema can be modi ed and/or deleted
150
F. Casati, M.G. Fugini, I. Mirbel
as needed by the designer. No special checks have to be performed, because the triggers, in this part of the tool, are not linked to the patterns of the catalog. Exception schemas can be modi ed, with respect to their exceptions. These modi cations include insertion of triggers, by searching in the catalog or by writing them from scratch, modi cation of the existing triggers, and deletion of some of them if they are no longer needed.
3.3 Schema Analysis Exceptions in WFs are typically designed individually, by identifying the exceptional situation, selecting the appropriate pattern from the catalog, and instantiating it in order to obtain a Chimera-Exc exception. However, although an exception considered individually may be correctly speci ed, mutual interactions among exceptions may lead to unforeseen and undesired behaviors. For instance, a set of exceptions may trigger each other, or two exceptions might operate on the same data, with the consequence that, if they are concurrently triggered, the nal result of their execution depends on their non-deterministic execution order. Exceptions analysis aims at statically (i.e., at compile-time) verifying that such undesired interactions do not occur. In particular, it aims at verifying two signi cant properties for a set of exceptions: termination and con uence [1, 3, 4, 20]. Termination holds if exceptions do not trigger each other inde nitely, so that rule processing eventually terminates. Termination analysis is typically based on building graphs depicting triggerings and activations among exceptions, and on detecting cycles in these graphs (see [4, 7] for details). Con uence requires the nal e ect of the rule processing (represented by direct or indirect changes to the WFMS database) to be independent on the (non-deterministic) scheduling of triggers having the same priority [8]. Sucient conditions for con uence have been proposed in [1, 3, 8]. Unfortunately, the above properties are undecidable in the general case. A fully automated analysis of interactions between exceptions is generally dicult to achieve, since it requires to understand the semantics of conditions and actions. A simple, conservative solution consists in checking that exceptions query and operate on di erent classes. More sophisticated and less conservative techniques for rule analysis, which consider the semantics of conditions and actions, have been proposed in [3]. In addition to adapting techniques proposed for active database rule analysis, a tool performing exception analysis may take into account the context in which exceptions are declared; in our model exceptions are always active, and if two exceptions share the same triggering event, they are both triggered as the event occurs, regardless of which part of the case is currently in execution. Exceptions can be concurrently triggered even if they are raised by di erent events, since exception scheduling occurs periodically, and within two subsequent exception processings many events can occur and many exceptions can be triggered, making the analysis more and more complex. However, although exceptions are always
An Environment for Designing Exceptions in Workflows
151
triggerable regardless of the context where they are declared, it is a good design practice to constrain the exception (within the condition part) to be active only when the element in which they are declared (e.g., a task) is running. If this is done, then we can reduce the scope of the analysis to exceptions of tasks that may be concurrently active (besides WF- and WFMS-level exceptions), thus reducing the number of exceptions that need to be concurrently analyzed. This is a key issue in exception analysis, since sucient conditions for termination and con uence are rarely met when the analysis is extended to the globality of declared exceptions, particularly when conservative approaches are followed.
3.4 Tool Architecture This section presents the architecture and the functionalities provided by the tool and gives a usage example. The architecture is depicted in Figure 3. The de nition of an exception for a WF schema is performed through the Schema Editor module, based on reusing the available patterns. From within the Schema Editor, the designer can interactively search the catalog for a suitable generic pattern using the Exception Retriever module. Retrieved patterns are instantiated/personalized for the WF schema at hand, through the Exception Writer module, which can also be used to de ne new patterns or exceptions from scratch, if the available ones do not t the WF requirements. Exceptions de ned for a schema can be analyzed by exploiting the Schema Analyser module of . The Catalog Manager performs the catalog management functions that is, insertion, modi cation and deletion of patterns. A Visual C++ /Windows prototype of is available, operating in a Client/Server environment, in accordance with the design guidelines of the other tools [6].
werde
werde
wide
werde
Managing the Catalog: The pattern catalog is extensible by de ning new
werde
patterns and/or specializing the existing ones. To this purpose, provides maintenance functionalities to manage the catalog (Catalog Manager module) by allowing insertion of new patterns and modi cation or deletion of existing ones. It also allows to add/remove pattern categories and sub-categories. Pattern management:
{ Creation: Newly de ned patterns are stored in the catalog, under the proper
category/sub-category. Fields of the pattern speci cation are interactively lled in. The pattern template is speci ed by invoking the Exception Writer editor. Patterns keywords can be selected from a proposed list. Keywords in the thesaurus are automatically updated whenever new keywords are speci ed for newly de ned patterns. Guidelines can be inserted as textual information to be compiled o -line by the pattern designer.
152
F. Casati, M.G. Fugini, I. Mirbel SCHEMA ANALYSIS
Task-level Task-set level WF level
Schema Analyser
Creation Modification Deletion SCHEMA MANAGEMENT
Schema Editor Insertion Modification Deletion
Keyword-based search Classification-based search
Exception Writer
Pattern catalog
Exception Retriever
Catalog Manager Category Addition Removal
Pattern Creation Modification Deletion
CATALOG MANAGEMENT
Fig. 3. Architecture of the werde tool
{ Modi cation: An existing pattern can be modi ed, after being selected using
the Exception Retriever. The designer can then select the parts of the pattern that must be modi ed, and operate on them through dialog windows. In it is possible to add new sample usages to an existing pattern template, by de ning an \executable example" and by selecting the template of interest from the catalog. { Deletion: it is performed by selecting the pattern of interest, and the tool performs the basic integrity checks.
werde
An example of work session, dealing with the speci cation of the termination pattern shown in Figure 2, is presented in Figure 4. Insertion/removal of categories: Categories and sub-categories can be added to the catalog at each level. Removal of categories implies the deletion of all subcategories, provided that the designer has already cancelled all the corresponding patterns.
Using the Catalog:
Werde
guides the designer during the process of exception speci cation by providing both classi cation information, which facilitates the search in the catalog, and guidelines to enter values for pattern parameters and to compose patterns into more complex structures.
An Environment for Designing Exceptions in Workflows
153
Fig. 4. Creation of the termination pattern in werde A typical exception-design session consists in activating the Schema Editor module. The main functionalities that can be invoked from within this module deal with the creation, modi cation and deletion of exception schemas. The functionalities provided through the creation and modi cation of exception schemas are the following:
{ Retrieval, to access the catalog and search for suitable patterns. The two
search modalities presented above are implemented (keyword-based and classi cation-based search). Figure 5a shows the "Retrieve Option" screen where the various search modes are available. In particular, consider that the termination pattern is being searched by keywords, as depicted in Figure 5b. The result of the search appears in Figure 5c. { Insertion, to support the interactive de nition of exceptions starting from a retrieved pattern or from scratch. The instantiation consists in binding the pattern parameters on the basis of the semantics of the WF schema, by exploiting the syntax-driven capabilities of the Exception Writer module and the template interface facility of . This facility consists of a set
werde
154
F. Casati, M.G. Fugini, I. Mirbel
(a)
(b)
(c)
Fig. 5. Pattern retrieval. Selection of retrieval option (a), keyword speci cation for keyword-based retrieval (b), and presentation of the retrieved pattern (c)
An Environment for Designing Exceptions in Workflows
155
of dialog windows which drives the designer in entering the patterns elds, possibly enabling the visualization of the guidelines and usage examples associated with the template under manipulation. The dialog facilities of the Exception Writer support also the de nition of new exceptions from scratch. The instantiation of the termination pattern retrieved through the steps depicted in Figure 5 works as in the sample work session of Figure 6, where the pattern is instantiated to obtain the trigger missingPayment of section 2.1. The instantiation steps consist of activating the exception writer module, selecting the item to be instantiated (<wfName>W in the example of Figure 6), and overriding it with the desired Chimera-Exc condition (roomReservation(R)).
Fig. 6. Instantiation of the termination pattern
{ Modi cation, to modify exceptions inside exception schemas. { Deletion, to remove exceptions from exception schemas. { Analyze, to analyze the exceptions written for the current schema, using
werde
the Schema Analyser module. currently provides a basic set of the static analysis functionalities, by focusing on interactions within each pair
156
F. Casati, M.G. Fugini, I. Mirbel
werde
of exceptions. In , two triggers are considered as con icting if they are active simultaneously and contain incompatible actions; actions are incompatible if the order of their execution leads to di erent states of the WF management database [6]. A table of incompatible actions is maintained to support the analysis, which can be performed at three levels: task level, to check exceptions de ned for a given task; task set level, to select a number of tasks for which the exceptions must be analyzed; schema level, to check all the exceptions de ned in the schema. Anomalous situations detected by the Analyser are signaled to the designer who can activate the Exception Writer for editing the involved exceptions.
4 Concluding Remarks
werde
We have presented the environment supporting the design of exceptions in WF schemas. The environment is based on a catalog of WF patterns which are a generalized description of triggers which can be reused when designing a new WF schema. Exceptions model both abnormal situations and typical start/end/repetitions of operations in a WF. We have presented the structure of the pattern catalog and the mechanisms for pattern reuse. We have also described the related functionalities of the tools for exception design and catalog management. Currently, the catalog is populated with a set of patterns which have been constructed by analyzing a set of real cases provided in the WIDE project testbed. A set of primitives for pattern reuse has been fully speci ed in [8] and is being tested to check the quality of these patterns in terms of signi cance and accessibility for reuse. Particularly the sample usages of patterns stored in the catalog are proving to be an e ective means for guiding the designer in selecting the appropriate patterns, in instantiating them appropriately, and in inserting the exceptions correctly in the WF schema. The implementation of has currently completed most of the functionalities described in the paper. Some functionalities, such as pattern design and schema analysis, are being extended in order to achieve a more complete and e ective suite of functions. For example, for schema analysis, we plan to consider exception termination analysis and con uence analysis on rule sets, and to consider in more detail the semantics of actions and conditions. For pattern design, we plan to have functionalities which help the catalog administrator in abstracting exception skeletons from WF schemas in a guided way and storing new patterns in the appropriate categories with suitable links to the existing repository. In fact, the catalog is regarded as an extensible tool to be populated gradually with new categories of patterns to be de ned by examining further real WF applications. Further points of research regard the use of exceptions as a basis for WF dynamic evolution and the design of a user feedback mechanism for validation of the quality and e ectiveness of a given WF, using adaptive techniques such as those studied in [11].
werde
werde
An Environment for Designing Exceptions in Workflows
157
Acknowledgments: The authors are thankful to B. Pernici and S. Castano for common work and ideas and to the students who implemented the tool.
References 1. A.Aiken, J.Widom, J.Hellerstein. \Behavior of database production rules: termination, con uence, and observable determinism". Proceedings of SIGMOD, San Diego, CA, 1992. 2. S. Bandinelli, A. Fuggetta, C. Ghezzi, \Software process model evolution in the SPADE environment", IEEE Transactions on Software Engineering, December 1993. 3. E. Baralis, J. Widom, \An algebraic approach to rule analysis in expert database systems", Proceedings of VLDB 94. 4. E.Baralis, S.Ceri, S.Paraboschi. \Improving rule analysis by means of triggering and activation graphs", Proceedings of the 2nd International Workshop of Rules in Database Systems, Athens, Greece, 1995. 5. R. Bellinzona, M.G. Fugini, B. Pernici, "Reusing speci cations in OO applications", IEEE Software, vol. 12, n. 2, March 1995. 6. F. Casati, P. Grefen, B. Pernici, G. Pozzi, G. Sanchez, \WIDE work ow model and architecture", Technical Report, Politecnico di Milano, n. 96-050, 1996. 7. F. Casati, S. Ceri, B. Pernici, G. Pozzi, \Speci cation of the rule language and active engine of FORO v.1", WIDE Technical Report n. 3008-6, September 1997. 8. F. Casati, S. Castano, M.G. Fugini, I. Mirbel, B. Pernici. \Using patterns to design rules in work ows", Technical Report n. 97-065 of the Politecnico di Milano, November 1997. 9. S. Castano, M.G. Fugini, I. Mirbel, B. Pernici, \Work ow reference models", WIDE Technical Report n. 3015-2, June 1997. 10. S. Ceri, R. Ramakrishnan, \Rules in database systems", ACM Computing Surveys, Vol. 28, No. 1, March 1996. 11. E.Damiani, M.G. Fugini, E. Fusaschi, \ COOR: a Descriptor Based Approach to Object-Oriented Code Reuse", IEEE Computer Magazine, September 1997. 12. U. Dayal, M. Hsu, R. Ladin, \Organizing long-running activities with triggers and transactions", Proceedings of ACM SIGMOD, 1990. 13. D. Georgakopoulos, M. Hornick, A. Sheth, \An overview of work ow management: from process modeling to work ow automation infrastructure, distributed and parallel databases", Vol. 3, n. 2, April 1995. 14. C. Ghezzi, M. Jazayeri, D. Mandrioli, \Fundamentals of software engineering", Prentice-Hall Intl., 1991. 15. D. Hollingsworth, \Work ow management coalition, the work ow reference model", Technical Report n. TC00-1003, November 1994. 16. R.E. Johnson, "Frameworks = components + patterns", in Communications of the ACM, vol. 40, n. 10, October 1997 17. G. Kappel, P. Lang, S. Rausch-Schott, W. Retschitzegger, \Work ow management based on objects, rules, and roles", IEEE Data Engineering Bulletin. Special issue on WF Systems, vol. 18, no. 1, March 1995. 18. C.W. Krueger, \Software reuse", ACM Computing Surveys, vol.24, n.2, June 1992. 19. D.C. Rine, \Supporting reuse with object technology", IEEE Computer, Special Issue on OO Development and Reuse, October 1997.
158
F. Casati, M.G. Fugini, I. Mirbel
20. J. Widom, S. Ceri, \Active database systems: triggers and rules for advanced data processing", Morgan Kaufmann, San Mateo, CA, 1996.
Automating Handover in Dynamic Workflow Environments ? Chengfei Liu1?? , Maria E Orlowska2, and Hui Li2 1 2
School of Computing Sciences, University of Technology, Sydney, NSW 2007, Australia Department of Computer Science and Electrical Engineering, The University of Queensland, Qld 4072, Australia
Abstract. Workflow technology has been widely used in business process modelling, automation and reengineering. In order to meet the fastchanging business requirements, to remain competitive in the market, an enterprise may constantly refine the workflow models of its business processes. The most challenging issue in evolution of a workflow model is the handover of its running instances from the old specification to the new specification. Such a handover depends on the semantics of a workflow model as well as the execution information of its running instances. A handover policy, therefore, needs to be specified for this purpose. In this paper, we propose a simple yet effective handover policy specification language. Using this language, a designer can easily specify a handover policy which reflect exactly what a workflow administrator needs to react to when a workflow model evolves. Criteria for the correct specification of handover policies are also addressed. Finally, a framework for automating handover of workflow instances is presented.
1
Introduction
Recent years have seen widespread use of workflow technology in business process modelling, automation and reengineering. Next only to the Internet related technology and products, the workflow technology and products [5,11] are arguably the most influential new breed of software systems from the perspective of achieving significant impact on enterprises. Many enterprises have shifted their data-centric approach in the context of the information systems technology and solutions to a process-centric one. Workflow technology has matured to some extent, and current products are able to support a range of applications. However, many limitations remain in current workflow technology, especially for supporting more demanding applications and more dynamic environment. ?
??
The work reported in this paper has been funded in part by the Cooperative Research Centres Program through the Department of the Prime Minister and Cabinet of the Commonwealth Government of Australia. Work done partially while the author was at the Distributed Systems Centre, Brisbane, Australia
B. Pernici and C. Thanos (Eds.): CAiSE’98, LNCS 1413, pp. 159–171, 1998. c Springer-Verlag Berlin Heidelberg 1998
160
Chengfei Liu, Maria E Orlowska, and Hui Li
In a fast-changing environment, an enterprise may constantly refine its workflow models to remain competitive in the market, to meet customers’ new requirements, to change business strategies, to improve performance and quality of services, to benefit from changes in technology, etc. Two aspects are related to the evolution of a workflow model. First, the old specification of a workflow model needs to be changed to a new specification correctly. This is the static aspect of the evolution. Second, as a business process tends to be long lasting, whenever a workflow model changes its specification, there may exist a set of running instances of the old specification. How to handover these running instances is an interesting and challenging issue. Whether or not the running instances shall evolve according to the new specification and how they evolve depend on an evolution policy which is specific to the business process which the workflow model represents for. We call this evolution policy as a handover policy. Currently, workflow evolution has not been sufficiently supported by workflow products. Only very primitive policies are supported by some workflow products such as Forte Conductor [4]. Other workflow products such as InConcert [6] support dynamic workflow adaptation at the instance level [14]. Schema evolution has been widely addressed in the field of Object-Oriented Databases and Software Processes [1,16,7]. However, little work has been done in addressing the problem of workflow evolution, particularly the dynamic aspect. In their paper [3] Casati et al. have presented a workflow modification language that supports modification of a workflow model (schema). They have also discussed the case evolution policies and have devised three main policies to manage case evolution: abort – to abort all running instances and to start new created instances following new specifications, flush – to finish all running instances first and then to allow new instances to start following new specifications, and progressive – to allow different instances to take different decisions. Though the progressive policy is further discussed in their paper, the fine granularity of the case evolution policy specitications has not been addressed. We view a workflow model evolution as a process which consists of three steps: (1). to modify a workflow model from its old specification to its new specification. (2). to specify a handover policy for handing over the running instances of the workflow model to be evolved. (3). to apply the handover policy. A workflow model modification can be done by applying a series of modification primitives using a workflow model modification language. In our study, we focus on the handover policy which is formulated based on the old and new specifications of a workflow model. When specifying a handover policy, we assume that a specifier has knowledge of both old and new specifications of the workflow model as well as their difference. Step 1 and Step 2 are performed at build-time, while only Step 3 is performed at run-time. The rest of the paper is organized as follows. In Section 2, we brief specification of workflow models. In Section 3, we design a handover specification language for specifying what a workflow administrator needs to react to the running instances when a workflow evolution occurs. Correct specification of handover policies is addressed in Section 4. In facilitating handover of workflow
Automating Handover in Dynamic Workflow Environments
161
instances, a framework for implementing the handover specification language is presented in Section 5. Section 6 concludes the paper with indication of our future work.
2
Workflow Model Specification
As the specification of a handover policy for an evolution of a workflow model is based on the old and new specifications of the workflow model, we first review the workflow modelling work. Several workflow modelling techniques have been proposed in the literature [12,2,8,13], some of them even target the transactional aspects of workflows. In [13], a graphical workflow specification language is proposed for workflow conceptual modelling. As shown in Figure 1, the language includes four types of modeling objects: task, condition, synchronizer, and flow. The flows are used to link the first three types of objects to build the specification of a workflow model.
Task
Condition
Synchronizer
Flow
Fig. 1. Modelling Objects of Workflows
– Task – A task is a logical step or description of a piece of work that contributes towards the accomplishment of a workflow model. It can represent both automated activities and human activities. Tasks are performed by assigned processing entities. A workflow specification is basically used to specify the coordination requirements among tasks. Sometimes properties of tasks may also be specified in capturing more aspects (e.g., transactional aspects) of a workflow model. However, the actual semantics of tasks is beyond the scope of workflow specifications. – Condition – A condition is used to represent alternative paths in a workflow specification depending on a conditional value. – Synchronizer – At certain points in workflows, it is essential to wait for the completion of more than one execution path to proceed further. A synchronizer is used for this purpose. – Flow – A flow defines the connection between any two objects, other than flows, in the workflow specification. It shows the flow of control or data from one object to another. A workflow model can be represented as a workflow graph using these modelling objects, where nodes of the graph can be tasks, conditions and synchronizers and links of the graph are flows. Restriction is placed in constructing a correct
162
Chengfei Liu, Maria E Orlowska, and Hui Li
workflow graph. Only a limited yet relatively complete set of constructs are supported by the language. They are Sequential, Exclusive OR-Split (Alternative), Exclusive OR-Join, AND-Split (Parallel), AND-Join (Synchronization), Nesting, Iteration, Start/Stop. Besides, a set of correctness constraints of workflow graphs have also been identified and verification algorithms have been proposed for verifying the syntactical correctness of workflow graphs specified using this language [15,13]. For instance, all or none of the flows proceeding a synchronizer activate for all possible instances of a workflow. Only one or none of the flows leading to a task or a condition activates for all possible instances of a workflow. The first rule eliminates the possibility of a synchronizer deadlock. The second rule eliminates the possibility of an unintentional multiple execution. In this study, we use this graphical language to specify workflow models and assume that workflow graphs of both the old and new specifications of workflow models specified using this language are syntactically correct. As handover is difficult to make inside an iteration block, we treat an iteration block as a single task.
3
Handover Policy Specification
A handover policy is specified to handover current running instances of a workflow model which is to be changed. It is used to model the dynamic aspect of workflow evolution. In this section, we design a handover specification language. The objectives of the language is effective yet simple. As when a handover policy is applied to an evolution of a workflow model (i.e., from its old specification to its new specification), the running instances may be executing at any task of the old specification. What is worse, different instances can take different paths to the same task. Therefore, different instances may require different handover strategies. No matter how complex the situation can be, the language should be expressive enough for specifying all a workflow administrator wants to specify. Obviously, if all the possibilities need to be specified explicitly, it can be cumbersome, even not applicable to large workflow models. Therefore, simplification of specification must be considered. Fortunately, in practice, a workflow administrator is only interested in some key points where turning actions need to be taken. Using some default and grouping specification, the specification of a handover policy can be greatly simplified. In the following, we discuss the handover specification language. 3.1
Syntax of the Language
Associated with every workflow model evolution is one and only one handover policy. A handover policy is defined by a set of handover statements. Three handover aspects of a running instance are described in each handover statement: – current position – indicating current executing task of a running instance; – history – indicating the traversed paths of a running instance by conditional value;
Automating Handover in Dynamic Workflow Environments
163
– action – indicating the action to be taken. A BNF definition of the handover policy specification language is given below: ::= {} ::= <do clause> ::= ON <position specification> <do clause> ::= DO | IF DO [ELSE <do clause>]
Position specification A position of a running instance is specified by the current executing task of that instance. In general, the exact point that the scheduler can interact is the completion point of the executing task of an instance. For the purpose of simplified specification, multiple tasks can be grouped to share a common do clause. Two ways of grouping are used: tasks without order and tasks with order. Position specification is further defined as follows: <position specification> ::= | "{" {,}"}" | TO
Action specification In supporting handover of a running instance, two important actions must be supported. One action is rollback. It is used to semantically undo some work so that the running instance can comply with the new workflow specification. A destination task must be given for a rollback action. The other is change-over. It is used to migrate the execution of a running instance (or a path of it) to follow the new specification. A destination task may or may not be given for a change-over action. If a destination task is not specified in the change-over action, the task which has the same name as in the current (old) specification in the new specification is chosen as the default destination task. There is another action called go-ahead which may not be explicitly specified. If no handover statement is defined on a task, the default handover action after the execution of the task is going ahead. As such, the specification of a handover policy can be greatly simplified. Only the turning points need to be specified. This coincides how a workflow administrator behaves to cope with a handover. Action specification is defined as follows: ::= ROLLBACK TO | CHANGE OVER [TO ] | GO AHEAD
Conditional turnings Sometimes, a turning action at the current task is decided based on the history (i.e., the traversed paths) of a running instance. This is supported by the conditional turning by representing the history information in a conditional value. Besides the history information, other semantic information (e.g., time) can also be specified in the condition to facilitate flexible handover.
164
3.2
Chengfei Liu, Maria E Orlowska, and Hui Li
Handover Policy Examples
We use the PC assembling workflow example introduced in [3] to illustrate how a handover policy can be specified using our handover specification language. As shown in Figure 2, the old assembly process starts by preparing in parallel a cabinet (either a Tower or a Minitower) and a motherboard (including CPU and disk controller). Then the motherboard is inserted into the cabinet. After the FDD is inserted, a condition on Container is checked to determine whether a cdrom needs to be inserted. Finally, the hard disk and video ram are inserted. The assembly process changes with the requirement that the cd-rom is replaced by NiceLab’s cd-rom 4x and audio card. Based on the different decisions, different handover policies can be specified as follows.
Start
Start
Prepare Motherboard
Get cabinet
Insert CPU
Prepare Motherboard
Get cabinet
Insert CPU
Insert Disk Controller
Insert Disk Controller
Plug Motherboard
Plug Motherboard
Insert FDD 1.44 MB
Insert FDD 1.44 MB Container = Tower
Container = Tower
Container = Minitower
Container = Minitower Insert Cd-Rom
Insert AudioCard Nicelabs
Add 1.6 GB HD
Add 1.6 GB HD
Plug Video Ram
Plug Video Ram
Stop
Stop
Old specification
Insert Cd-Rom NiceLabs 4X
New specification
Fig. 2. Old and New Specifications of PC Assembling Workflow Model
Automating Handover in Dynamic Workflow Environments
165
Example 1. Specification of concurrent to completion policy, i.e., all running instances are allowed to terminate following the old specification, while new instances can be started following the new specification. This policy can be expressed by the following single statement. ON Start TO Plug-Video-Ram DO GO AHEAD.
Example 2. Specification of Abort policy, i.e., all running instances are aborted, and the newly created instances will start following the new specification. ON Start TO Plug-Video-Ram DO ROLLBACK TO Start.
Example 3. Change over the running instances if executing before the condition checking on Container, otherwise go ahead. ON Insert-FDD-1.44M DO CHANGE-OVER.
Example 4. Change over before the condition checking on Container if the instance will satisfy the condition Container = Tower ; Rollback to the end of the task Insert-FDD-1.44M if the instance takes the path affected by the change. ON Insert-FDD-1.44M IF Container = Tower DO CHANGE-OVER; ON Insert-CD-Rom DO ROLLBACK TO Insert-FDD-1.44M; ON {Add-1.6GB-HD, Plug-Video-Ram} IF Container = Tower DO ROLLBACK TO Insert-FDD-1.44M.
As shown by above examples, handover policies can be easily and directly specified using the handover specification language. A specifier only needs to explicitly specify the turning actions taken at turning points. In Example 4, a handover policy which consists of three explicit handover statements is specified. The instances executing along the path which is not affected by the change take default action, i.e., go-ahead. This example shows that arbitrary handover policies can be specified using the handover specification language.
4
Correctness Issue of Handover Policy Specification
With a handover specification language, workflow specifiers have the flexibility to support fine-granularity of handover policies. However, it may also bring the errors into specifications. As the correctness checking of a workflow model specification, it is also important to check whether a handover policy is specified correctly. As the specification of a handover policy is different from the specification of a workflow model, new correctness problems may exist for a handover policy specification. In order to study the correctness issues of handover policy specification, we first introduce a so-called handover graph. Each handover policy can be defined
166
Chengfei Liu, Maria E Orlowska, and Hui Li
by a handover graph. The handover graph is constructed based on the workflow graphs of both old and new specifications. Each handover statement is reflected in the handover graph as follows: (1). If a rollback action is defined on a task T , add a link from T to the task to which it rolls back; add a special dead task Td and move all links originated from T to Td . A dead task is a task which never gets executed. (2). If a change-over action is defined on a task T , add a link from T to the task to which it changes over; add a dead task Td and move all links originated from T to Td . (3). If a (default) go-ahead action is defined on a task T , keep the workflow graph of the old specification unchanged for T . (4). If a conditional turning is specified, add a condition task Tc with two links originated from Tc specifying the two exclusive condition values. Depending on the turning action change the graph accordingly. Example 5. The handover graph for the policy defined in Example 4 is given in Figure 3.
4.1
Syntactical Error Types
As the handover graph for a workflow model evolution is constructed based on workflow graphs of the old and new specifications of the workflow model and these workflow graphs are assumed syntactically correct, a handover graph can be erroneous only if the turning actions are specified incorrectly. Therefore, we aim at errors resulted from incorrect specification of turning actions. The error types which we have identified in specifying a handover policy include: – cyclicness If a cycle appears in the handover graph of a handover policy, the running instances in the cycle will execute endlessly. Such a cycle must be avoided during specification of a handover policy. In Example 4, if we change the condition of either the first or the third handover statement to Container = M initower, a cycle will occur and the execution will never stop. – deadlock For a synchronizer node of the handover graph of a handover policy, if a dead task appears in one incoming path but does not appear in another path, a deadlock problem exists for the handover policy specification. For example, if we change over one branch of a parallel construct while keeping another branch go ahead, a deadlock will occur. Another example of deadlock can be resulted from structure mismatch. If we want to change over branches of an alternative construct to a parallel construct, a deadlock will happen to the new specification. – unintentional multiple execution Similar to a workflow model specification, an unintentional multiple execution error may occur in a handover policy specification. One such example
Automating Handover in Dynamic Workflow Environments
Start
167
Start
Prepare Motherboard
Get cabinet
Insert CPU
Prepare Motherboard
Get cabinet
Insert CPU
Insert Disk Controller
Insert Disk Controller
Plug Motherboard
Plug Motherboard
Insert FDD 1.44 MB
Insert FDD 1.44 MB
Container = Tower
Container = Tower
Container = Minitower
Container = Tower
Container = Minitower
Insert AudioCard Nicelabs
Insert Cd-Rom Dead Task
Insert Cd-Rom NiceLabs 4X
Add 1.6 GB HD
Add 1.6 GB HD Plug Video Ram
Container = Tower
Plug Video Ram
Stop
Container = Tower Stop
Fig. 3. A Handover Graph may come from changing over multiple parallel branches directly or indirectly to the same task of the new specification. Another example can be changing over different branches of a parallel construct to different branches of an alternative construct, respectively. 4.2
Correctness Criteria for Handover Policies
In preventing handover policy specification errors, we define a set of correctness constraints as follows. Rule 1 At most one handover action may be executed for each task of a running instance. In other words, if a cycle appears in the handover graph of a handover policy, then the conjunction of conditions specified in all condition nodes along the cycle must be false, i.e., the cycle is a pseudo cycle and no real cycle is allowed in the handover graph.
168
Chengfei Liu, Maria E Orlowska, and Hui Li
Rule 2 Either all branches or no branches of a parallel construct of the old specification are changed over to new specification. In other words, either all dead tasks or no dead tasks can be connected to a sychronizer. Rule 3 Branches of an alternative construct in the old specification cannot be changed over to branches of a parallel construct in the new specification. Rule 4 Branches of a parallel construct in the old specification cannot be changed over to branches of an alternative construct in the new specification. Rule 5 Branches of a parallel construct in the old specification cannot be changed over directly or indirectly to the same task of the new specification. These rules are verified for each handover policy specification before it is applied to running instances, thus run-time handover errors can be greatly reduced.
5
Facilitating Handover Policies
As automatic handover of running workflow instances has not been addressed by existing workflow management systems, it is ideal to put forward a framework which can facilitate handover based on current workflow technology. In this section, we address some key technical points towards such a framework. 5.1
Required Data Structures
The data structure for workflow instances is designed as follows: W F Inst(InstID, State, History, ActiveP ath(SpecID, CurrentP osition)) Where InstID is used for identifying a workflow instance. State records the current state of the workflow instance, such as, executing, completed. Two additional states migrating and migrated are introduced for handover purpose. The migrating state indicates that the workflow instance is under a handover process, when the handover process is finished, the state of the workflow instance is changed to the migrated state (not turning back to the executing state). The migrated state indicates that the instance needs to be treated specially in case rollback or another handover (due to newer specification or version of its workflow model) may be required to the instance later since it has undergone a handover process. History records the log data of all traversed paths of the workflow instance. A workflow instance may contain several active parallel paths. The number of paths will increase after an AND-Split is reached and will decrease after a synchronizer (AND-Join) is reached. Every active path has a SpecID and a CurrentPosition associated with it. A SpecID is used for identifying the specification on which
Automating Handover in Dynamic Workflow Environments
169
the execution of that active path is based. During handover, it is possible that one active path is running on the old specification while another is running on the new specification. A CurrentPosition records the currently executing task of that path. In addition, a new data structure for policy specification is designed. P olicy(SpecID, N ewSpecID, T urning(T ask, Condition, Action, Destination)) Where SpecID and NewSpecID are used for identifying the old and new workflow specifications on which a handover policy specification is based. As one and only one policy is associated with each workflow model evolution, a policy can be identified by a SpecID. Several turnings can be described in a policy. Each Turning records a task – after its completion the turning will take place, a condition – the turning may take effect only under the condition, an action – either rollback or change-over, and a destination – indicating the destination task of the turning. 5.2
Applying a Handover Policy
To apply a handover policy to a workflow specification, a workflow system command can be issued: handover(SpecID) This command will automatically change the state of all running instances of the specification indicated by SpecID to the state Migrating. 5.3
Scheduling a Handover Action
When a task t indicated by CurrentPosition in an active path p of a running workflow instance w finishes its execution, the scheduler will schedule the next step for executing. Two cases are scheduled differently: (1) If the instance w is in a state other than Migrating, the scheduler will schedule w (specifically the path p) as usual, i.e., to take the next step from t according to the specification of p indicated by SpecID. The scheduling information is recorded in the History. (2) If the state of the instance w is Migrating, the scheduler will first find the policy defined on the specification of p indicated by SpecID. After that, it tries to match t with the Task in a Turning of the policy. if it matches one and the Condition in the Turning is satisfied, then take the Action in the Turning. Otherwise keep trying. If there is no one matches, take the default next step as usual according to the specification of p indicated by SpecID. Taking a Rollback Action If a rollback action is scheduled, the path p of the workflow instance w is rolled back to the point specified by Destination. The rollback process may be undertaken with the help of partial compensation [8,10,9]
170
Chengfei Liu, Maria E Orlowska, and Hui Li
which is another technical issue in workflow management systems. The information of the rollback action and all the rollback steps need to be recorded in the History. After the rollback action is completed, CurrentPosition is changed to hold the Destination. Taking a Change-Over Action If a change-over action is scheduled, the path p of the workflow instance w is changed to run on the new specification. The point where the path continues in new specification is specified by Destination. The information of w needs to be modified as follows: (a) SpecID for the path p is changed to hold the new specification ID, i.e., NewSpecID. (b) CurrentPosition for the path p is changed to hold the Destination. (c) A log item indicating the change-over action needs to be added to the History. (d) If all active paths of the workflow instance w are changed over to running on the new specification, the state of w is changed to Migrated and the state change is also recorded in the History.
6
Conclusion and Future Work
It is an important yet challenging topic to handover running workflow instances for enterprise computing. In this paper, we specifically addressed this topic. To support flexibility and fine-granularity of handover policy specification, we designed a simple yet effective handover policy specification language. Using this language, a designer can easily and directly specify a handover policy which reflect exactly what a workflow administrator needs to react to when a workflow model evolves. Correct specification of handover policies was also discussed. A framework for automating handover of running workflow instances was presented. In the future, we will investigate algorithms for verifying the correctness of handover policy specifications. As well we will extend our current two-version support of workflow models (i.e., old specification and new specification) to multi-version support (therefore multiple handovers of long-running instances).
References 1. J. Banerjee, W. Kim, H-J. Kim, and H. Korth. Semantics and implementation of schema evolution in object-oriented databases. In Proceedings of the ACM SIGMOD International Conference on Management of Data, pages 311–322, 1987. 2. F. Casati, S. Ceri, B. Pernici, and G. Pozzi. Conceptual modeling of workflows. In Proceedings of OO-ER conference, pages 341–354. Lecture Notes in Computer Science, Vol. 1021, Springer, 1995. 3. F. Casati, S. Ceri, B. Pernici, and G. Pozzi. Workflow evolution. In Proceedings of the 15th ER Int. Conf., pages 438–455. Lecture Notes in Computer Science, Vol. 1157, Springer, 1996. 4. Forte. Forte Conductor Process Development Guide. Forte Software, Inc., 1994. 5. Butler Group. Workflow: Integrating the Enterprise, June 1996.
Automating Handover in Dynamic Workflow Environments
171
6. InConcert. InConcert Technical Product Overview. InConcert Inc., January 1997. 7. M. Jaccheri and R. Conradi. Techniques for process model evolution in EPOS. IEEE Transactions on Software Engineering, 19(12):1145–1156, December 1993. 8. D. Kuo, M. Lawley, C. Liu, and M. Orlowska. A general model for nested transactional workflows. In Proceedings of the International Workshop on Advanced Transaction Models and Architectures, pages 18–35, 1996. 9. F. Leymann. Supporting business transactions via partial backward recovery in workflow management systems. In Proceedings of BTW’95, pages 51–70, 1995. 10. C. Liu and M. Orlowska. Confirmation: Increasing resource availability for transactional workflows. Technical report, Distributed Systems Technology Centre, September 1997. 11. Workflow Management Coalition Members. Glossary – A Workflow Management Coalition Specification. Workflow Management Coalition, November 1994. Softcopy available via: http://www.aiai.ed.ac.uk/project/wfmc/. 12. M. Rusinkiewicz and A. Sheth. Specification and execution of transactional workflows. In W. Kim, editor, Modern Database Systems: The Object Model, Interoperability, and Beyond. Addison-Wesley, 1994. 13. W. Sadiq and M. E. Orlowska. On correctness issues in conceptual modeling of workflows. In Proceedings of the 5th European Conference on Information System, 1997. 14. A. Sheth and K. Kochut. Workflow applications to research agenda: Scalable and dynamic workflow coordination and collabration systems. In Proceedings of the NATO ASI on Workflow Management Systems and Interoperability, August 1997. 15. A.H.M. ter Hofstede, Maria E. Orlowska, and J. Rajapakse. Verification problems in conceptual workflow specification. In Proceedings of the 15th ER Int. Conf., pages 73–88. Lecture Notes in Computer Science, Vol. 1157, Springer, 1996. 16. R. Zicari. A framework for schema updates in an object-oriented database systems. In Proceeding of 7th International Conference on Data Engineering, pages 2–13, 1991.
Document-Centric Groupware for Distributed Governmental Agencies Daniel A. Tietze*, Ajit Bapat*, Rolf Reinema** GMD – German National Research Center for Information Technology * Integrated Publication and Information Systems Institute (IPSI) ** Institute for Telecooperation Technology (TKT) {daniel.tietze | ajit.bapat | rolf.reinema}@gmd.de
Abstract The distribution of the German government between Bonn and Berlin calls for the technical support for the collaborative document-based tasks performed by inter- and intra-ministerial work groups. The currently available cooperation and communication tools do not provide an integrated and homogeneous cooperation environment suitable for this application domain. This paper presents the Cooperative Document Repository, an integrated document-centric collaboration environment. The integration of cooperative document management and document processing with multi-point audio/video conferencing provides a flexible collaboration environment within which a wide range of collaborative activities can be performed. Keywords: groupware; CSCW; document-based collaboration; federal government; distributed administrations; POLIWORK; video conferencing
Introduction The German unification and the resulting move of governmental agencies from Bonn to the new capital, Berlin, will result in a distribution of administrative units between Bonn and Berlin. This distribution will be both along vertical as well as horizontal lines, meaning that some ministries will be moved entirely to Berlin, some will remain in Bonn, while others will be distributed between Bonn and Berlin. As a result, work groups and task forces (both within a ministry as well as between ministries), which were previously co-located will be distributed as well and will require technical support in order to continue their work. To address the challenges of this distribution and to provide technical solutions and organizational suggestions for the resulting distributed work processes, the German government instated the POLIKOM research program in 1994, within which four projects are addressing the different issues arising from the close cooperation between distributed governmental agencies. B. Pernici and C. Thanos (Eds.): CAiSE’98, LNCS 1413, pp. 173-191, 1998. Copyright Springer-Verlag Berlin Heidelberg 1998
174
Daniel A. Tietze, Ajit Bapat, Rolf Reinema
At GMD, the authors are involved in one of these projects: POLIWORK1 – Telecooperation and cooperative document management at the personal workplace – which began in 1994 and will continue until the end of 1998. The focus of this project is to support the cooperative tasks of the governmental agencies, specifically those collaborative activities centering on document management and document processing, both in synchronous as well as in asynchronous settings, through the development and installation of appropriate support technology. The user community of the project consists of two German federal ministries, the Ministry of the Interior (BMI) and the Ministry of Economics (BMWi). This paper describes the central software component developed for the user community: the Cooperative Document Repository (CDR), an integrated collaboration support system which inter- and intra-ministerial work groups can use in order to structure and store the documents involved in their cooperative work processes as well as to initiate the required communication and collaboration tasks. The remainder of the paper is organized as follows. The next section presents brief scenarios of collaborative activities found in the user community. From these a set of requirements is derived to be addressed in the subsequent system design. The architecture and functionality of the CDR are then presented, followed by a description of the approach of document-centric collaboration support. The paper concludes with a comparison to related work in the field of collaboration support technology and a brief outlook on future work.
System Requirements The requirements for the technical solution which are presented in this section were gathered in the user community through an analysis of the cooperative work processes found as well as through interviews with the users. The resulting work process scenarios were analyzed in order to determine how to adequately support the arising tasks by introducing a collaboration support system combining state-of the-art communication hardware with advanced groupware technology. Also, the existing infrastructure found within the ministries was examined in order to determine in which way it could be integrated into a collaboration support environment. Collaborative Activities In the scope of the project, we concentrated on two main scenarios for cooperative activities in the ministerial environment: joint text creation by a work group as well as presentation and discussion including the higher levels of the ministerial hierarchy.
1 The work presented in this paper is partially funded by the German Federal Ministry of Education, Science, Research and Technology (grant number 01 IT 403 A-D).
Document-Centric Groupware for Distributed Governmental Agencies
175
A typical task in the application field is the preparation of a legislative draft. Usually, such a text is created jointly by a small group of people. Different parts of the document are created by different authors and then integrated into one common document. The various intermediate versions of the draft are passed around among the members of the work group. Paper documents are distributed using fax or regular mail while documents in electronic form are sent via e-mail. To coordinate the integration of the various parts bilateral communication (e.g. telephone call) as well as multilateral communication (e.g. meeting) takes place. Meetings include a certain amount of preparation (setting a date, distributing the necessary documents, etc.) as well as wrapping up (writing minutes etc.). Another scenario is closely related to the one described above. At the end of the joint text creation the resulting draft needs to be approved by the higher levels in the ministerial hierarchy. For this, the final draft is passed to the higher level and then, in a meeting with the superior the draft is presented and discussed, changes to be made are annotated. Afterwards, the document is revised accordingly. Requirements Based on the work processes found in the ministerial context, a number of requirements can be identified for a system that is to support the distributed, cooperative tasks in the application field. (R1) All work processes are document-based. Either texts are created and edited as part of the process or the process involves the presentation, discussion and annotation of documents. Therefore, document-based support is indispensable. This requirement combines several important aspects: Documents need to be integrated directly into the work processes, including the support of existing document bases (R1.1). In order to work on the documents, cooperative document processing is required. One important aspect of this is the support of legacy applications (R1.2), since the ministries have a certain set of document processing tools which they regularly use (e.g. one of the established office suites). Documents have to be passed to other people involved in the task. Thus, document exchange is to be supported (R1.3). Furthermore, storage of the documents is necessary (R1.4). And finally, paper-based documents need to be integrated (R1.5). (R2) The second important aspect is the provision of communication channels between the people involved in the work processes. Synchronous communication (R2.1) and the high quality of the communication channels (R2.2) are the major requirements. Since the collaborative processes span multiple sites, the communication support should not only be bilateral but also support multi-party distributed conferencing (R2.3). (R3) One key aspect with respect to the acceptance of the system by the users is the usability of the supporting system. To achieve this, several aspects have to be considered: A homogeneous user interface (R3.1), integrated access to various functionalities of the system (R3.2), as well as a familiar look and feel of the system’s user interface (R3.3).
176
Daniel A. Tietze, Ajit Bapat, Rolf Reinema
(R4) Due to the fact that the system is to be used in the Federal Ministries the existing technical infrastructure found there needs to be taken into account. The ministries are connected by an ISDN network which is physically separate from the public telephone and ISDN networks and therefore regarded as being secure. Thus, communication should run over these lines (R4.1). In addition, the government is setting up a governmental IP backbone that connects all ministries within and among the two cities of Bonn and Berlin. Again, this network is to be used (R4.2). Additionally, the system that is to be developed should run on the governmental intranet which is currently being installed (R4.3). (R5) In addition to the previous requirements, some general requirements result from the fact that the system is developed in a pilot project. Adaptability to other user fields, e.g. other ministries (R5.1), as well as scalability to larger numbers of users (R5.2) are required.
System Design In order to ensure rapid availability of the key functionalities required in this project a major design goal was to take already existing components from the market. By doing so, an initial set of requirements could be fulfilled by off-the-shelf components. These application modules then needed to be integrated into the system in a way which makes the combined functionality of the tools easily available to the users. A first version of a collaboration support environment has already been installed at an early stage so that the users could become familiar with the new possibilities offered by communication and cooperation technology. Support for joint viewing and editing of documents by two or more participants is provided by the use of a standardized application sharing package which enables several users at the same time to jointly view and interact with any application and thus any kind of document. In this way, the standard application suites and legacy applications which were already used within the ministries as well as the documents produced by these are integrated into the collaborative processes in a familiar and easy to use manner. In addition to this the use of a color flatbed scanner made it possible for the users to import paper based documents and enabled them to work on them. Both the sharing of any application between several sites as well as the possibility to import any paper-based document fulfilled the requirement for joint viewing and editing of existing document bases. As already mentioned above, the users have asked for high quality synchronous audio/visual communication while working together on documents. This resulted in the provision of off-the-shelf components for audio/video and data conferencing. The two user communities within the project, the German Federal Ministry of Economics (BMWi) and the Federal Ministry of the Interior (BMI) have had different needs for communication and collaboration support [1]. While the BMWi has expressed the need for desktop-based collaboration between single users in the form of desktop workstations equipped with conferencing and collaboration
Document-Centric Groupware for Distributed Governmental Agencies
177
technology, the BMI has asked for meeting rooms to let groups of users from within the ministry or between different ministerial agencies collaborate with each other. So they were provided with a number of electronic meeting rooms (so called team rooms) equipped with large projection units coupled to the same kind of workstations as already used for the desktop users. Different configurations achieved in the initial solution are shown in figure 1.
178
Daniel A. Tietze, Ajit Bapat, Rolf Reinema
Desktop Workplace Equipped with multimedia PC including A/V Conferencing and document processing. Scanner and printer allow import and export of documents in paper form.
Desktop Workplace ISDN Connection 3S0 (384 kBit) or 1S0 (128 kBit) Used for A/V Conferencing and data conferencing.
Governmental IP Backbone When available, will be used for Data Conferencing with data sharing or collaborative tools.
Team Room Multipoint Conferencing Unit (MCU). Facilitates ISDN-based A/V conferences between multiple sites.
Team Room Additionally equipped with projection unit for near-life-size displays of conference participants (replaces second display screen)
Team Room Uses two monitor screens for independent display of conference picture and data conferencing applications.
Team Room
Figure 1 : Variations of technical solution
As a basis for the audio/visual communication there were two different kind of networks: a governmental owned ISDN network (BBN) as well as a governmental owned IP-network (IP-Backbone). Because of the fact that currently every ministry has access to the BBN but not to the IP-Backbone, ISDN has been chosen as the basis for synchronous communication. A further and by far more important reason was the fact that ISDN provides guaranteed bandwidth and delay which are
Document-Centric Groupware for Distributed Governmental Agencies
179
necessary to provide the audio and video quality the users have asked for. For the desktop solution, a bandwidth of 128 kbps was deemed appropriate, while the large projection unit in the team rooms, together with the requirement of hosting discussions between a number of users at each site required higher bandwidths (384 kbps), in order to provide better video and audio quality for all users in the room. Hardware Codecs with integrated ISDN connectivity are used in order to achieve high performance. Their compliance with the international ITU-T standards H.320 [2] and T.120 [3] ensures interoperability with ISDN-based conferencing systems from other vendors. In order to provide the required multi-point communication, an MCU (Multi-point Control Unit) has been installed which is able to handle a large number of simultaneous conferences, each between several sites. The MCU manages multipoint conferences by distributing the A/V data between the connected sites. It integrates multiple A/V data streams into a single stream for each site which could, for instance, display multiple sites on one screen, e.g. in the form of a quad-split display showing four remote sites at the same time. Assessment of Initial Solution After the installation and setup of the off-the-shelf components some of the previous mentioned needs of the users could already be met, but there were still several remaining requirements which necessitated the development of the system presented in this paper. With the provided video conferencing system it became possible for all users (even for those who had no connection to the IP-Backbone) to exchange documents in electronic form faster than before. But what the users were missing was a document repository into which they could file their documents for retrieval within as well as between collaborative sessions. The repository should allow structuring the document base by work groups and restrict the access to those work groups or task forces. Many of the necessary steps to establish collaboration between two or more users are purely administrative or consist of application access and application control, tasks which the integrated collaborative application could easily perform for the users. Additionally, many of the necessary steps are performed by different applications, each providing its own distinct user interface, forcing the users to be familiar with a large number of applications just in order to perform the task at hand. Instead, the envisioned system should be provided with enough information beforehand, so that arranging a collaboration session between a number of users would be ideally reduced to merely selecting the document(s) to be cooperatively processed and the user(s) with whom to collaborate. All administrative tasks, the scheduling and reservations as well as the timely invocation of the required tools were to be controlled by the system, thus giving the users the freedom to concentrate on the main task at hand: the joint processing and creation of documents.
180
Daniel A. Tietze, Ajit Bapat, Rolf Reinema
A major requirement of the users was the provision of a single integrating user interface to control all the different functional modules of the application suite. The off-the-shelf components were neither integrated nor did they provide a homogeneous user interface with a familiar and consistent look and feel to the users. A general problem with today’s MCUs is that they do not offer any kind of (standardized) API (Application Programmers Interface), rather they require an operator who carries out conference reservation and control instead of the users. This resulted in the requirement to develop a conference reservation and control system which allows users to create and control audio/video and data conferences themselves. Another problem with today’s MCUs is that ad-hoc communication is almost impossible, because of the way in which an MCU conference is set up: When a user decides that he needs a multi-point conference he has to ask an MCU operator to set it up at a certain day and time. This can be very time-consuming (it may take several minutes to hours depending on the availability of the operator). Also, the way an MCU controlled conference is typically set up results in the problem that a multi-point audio/video conference can not be resized dynamically in terms of time and number of participants, i.e. during a running conference, no further users can participate unless that their later participation was already anticipated when making the necessary reservation. An analysis of the remaining unmet requirements led to the specification and development of a Cooperative Document Repository (CDR), into which the initial components were to be integrated and which provided a consistent and easy to use interface to the available functionalities.
Architecture of the Cooperative Document Repository The CDR is a distributed collaborative system, providing the users with a cooperatively accessible document storage as well as immediate information about other users' actions. The architecture is centered around a database server which provides persistence of documents and data objects as well as propagation of realtime update notifications. This central database is provided using a commercially available RDBMS accessed via the ODBC [4] database access layer. The use of a standard RDBMS and ODBC as an access interface was chosen in order to meet the adaptability requirement, as it does not limit the system to a specific DBMS and thus ensures applicability in a wide range of user communities. The central database stores all information required by the system: the document management metadata, the infrastructure and group modeling data as well as the actual document contents. For the Cooperative Document Repository a central data model was developed, based on previous work on a file and document model for the public administration [5] performed within the project. The original object model was revised to address the specific requirements introduced by the collaborative settings to be supported by the project’s application goals, especially the stronger
Document-Centric Groupware for Distributed Governmental Agencies
181
need for group and access right modeling. The CDR object model contains all relevant information about not only the contents of the document repository, but also about the system environment, the users accessing the repository and the conferencing equipment used at the different sites. The client applications of the distributed system were developed in Java, using the JDBC [6] interface as a means to access the database server. Java was chosen as an implementation language in order to make use of the existing and future infrastructure within the ministries. The use of the Java programming language and the JDBC database access mechanisms ensures operability of the solution over the governmental IP backbone and also enables, at a later stage, the integration of CDR modules as general services into the currently evolving governmental Intranet. The use of the standard database access mechanism JDBC provides a large degree of flexibility in the selection of the database system used within the system as well as the actual driver library used for database access. The database access module is configurable in order to allow easy exchange of database drivers, which also aids the evaluation of the performance characteristics of the different access libraries. As an example for this flexibility, two different JDBC packages were used interchangeably and evaluated in the course of the development of the CDR. The truly cooperative use of the CDR in the context of a synchronous audio/video conference necessitates the synchronous coupling between the users' client applications. This is performed through instantiation of the CDR object model in the clients in the form of a distributed shared memory. The clients' data sets are automatically kept consistent through the use of the underlying database's transaction system and a combination of object invalidation and active notifications. Client side operations are gathered into transactions, communicated to the database server, whose responsibility it is to ensure transactional consistency. Objects modified within a transaction are identified and an active notification component, based on trigger mechanisms available in the database server, automatically distributes the changes to all connected clients by invalidating the locally cached object replicas and notifying the client applications of these changes. The automatic notification leads to a redisplay of those changed data objects which are currently displayed on the user's screen, thereby providing interactive synchronous collaboration. The client applications contain interfaces to external components such as A/V conferencing, application sharing and integration of a scanner for documents in paper form. Many of the off-the-shelf components used in the project, such as the application sharing system or the videoconferencing system, merely provide C++ programming interfaces which had to be made available to Java applications in a consistent manner. In order to ease flexibility and exchange of components at a later stage, generic Java interface modules were specified for the individual functional modules which abstracted from the actual programming API and presented the module’s functionality in an abstract and reusable manner. These interface modules were then mapped onto the interfaces available in the application
182
Daniel A. Tietze, Ajit Bapat, Rolf Reinema
modules currently used. This abstraction layer fulfills the adaptability requirement, since it effectively hides the set of third-party cooperation and communication tools actually used and facilitates easy exchange of system components. In order to fulfill the requirement for supporting existing document processing applications and enable the cooperative work on documents created and maintained with non-cooperative applications such as the familiar office suites, the CDR provides an integration with external applications (e.g. a text processing system) as well as with third-party application sharing packages. This is achieved through an abstract application integration layer which can be tailored to support a wide range of applications for different document types as well as a number of application sharing packages, accessed through their API functionality. The integration with the application sharing package allows automatic launching of the appropriate application along with the application sharing module within a conference. Due to its knowledge of the current interaction situation and the set of connected users, the CDR can launch the relevant tools and applications in a mode which is required for the current communication setting. To support the requirement of integrating paper-based documents into the cooperative work processes, scanners and printers are integrated into the CDR. Documents can be scanned and directly imported into the CDR by way of a TWAIN compatible scanning device, thus making them directly available to the other collaboration partners. For multipoint conferences a reservation has to be made on the MCU. The functionality for making such an MCU reservation is integrated into the CDR. The control and scheduling of the Multipoint Conferencing Unit is performed via the MCU's reservation database, since the MCU itself does not offer a directly usable API. For this task, a specialized daemon was developed which mediates between the MCU's reservation database (resident on a dedicated server machine) and the CDR database. Any relevant changes performed in the CDR database (e.g. the scheduling of a new conference) is immediately fetched and transferred to the MCU reservation database. This daemon interfaces to the CDR server just like the other clients, by way of the distributed shared memory which is automatically kept in a consistent state when changes occur, thereby informing the daemon that changed data needs to be propagated to the MCU reservation database. The architecture diagram in figure 2 shows the "building blocks" of the CDR application. The shaded boxes denote the modules developed in Java, boxes with emphasized borders denote individual applications within the CDR.
Document-Centric Groupware for Distributed Governmental Agencies
C D R F rontend (G U I)
S hared O bjects
JD B C D river
A pplication Integration Interface
E xternal A pplications
A pplication S haring Interface
A pplication S haring T ool
S canner Interface
S canner T ool
AV C onferencing Interface
AV C onferencing T ool
C lient(s)
O D B C D river
JD B C D river
S hared O bjects
O D B C D river M C U C ontrol and R eservation O D B C Interface D atabase S erver
D ocum ent D atabase
JD B C D river O D B C D river
Server
Schem a Legend : M odule developed in Java : Individual A pplication
MCU R eservation D atabase
M CU Reservation Server
Figure 2: Architecture Diagram of CDR
183
184
Daniel A. Tietze, Ajit Bapat, Rolf Reinema
Usage and Functionality of the Cooperative Document Repository The repository front-end is a group-aware application: it not only provides the users with a view of the repository's contents, but also displays information about which users are currently cooperatively viewing or editing which documents. This groupawareness is provided in the form of name tags added to the repository content displays, indicating current cooperation activities on the documents (see figure 3). In order to be immediately applicable by the users, the CDR provides a document management paradigm similar to the file management structures already familiar to the users from the Windows environment. The CDR enables the users to structure the cooperatively used document space into individual repositories, containing a hierarchy of folders, which in turn contain the documents. The access rights and group modeling mechanisms present in the CDR allow the users to control and manage their group cooperation and restrict other users’ access to the documents in the CDR. Every user of the Cooperative Document Repository can create groups and repositories to match his or her current requirements. For a task which is to be tackled cooperatively, a new group within the system can be created, including those users which require access to the common documents. Along with this, a new repository can then be set up, using group- and user-based access permissions to control which users and which user groups may view or edit the documents contained in the repository. The newly created repository (or, rather, an icon depicting it) automatically appears on the CDR desktops of all users who are allowed to access the repository and who are currently logged into the system, allowing them to begin their work immediately. The solution is kept scalable to a large number of users by only presenting those repositories and repository contents at the GUI to which the current user has access. The group members can now cooperatively begin to structure and fill the repository, by creating folders as required and importing documents relevant for the task at hand. In this way, the group members are provided with a common, cooperative document storage system which they can use to store and exchange documents related to the task at hand. The fact that all users have concurrent access to the CDR simplifies exchange of documents.
Document-Centric Groupware for Distributed Governmental Agencies
185
Figure 3 : Desktop view of CDR
It is important to note that the CDR was not intended to be a replacement for classical document management and storage systems. All tasks involved in the strict document management methods required in the ministerial context, such as registering all documents entering and leaving the ministry, managing consistent document and file numbers, maintenance of a filing plan, etc. are outside the scope of the CDR. All official document management functionalities will continue to be provided by dedicated document management systems (DMSs), which are approved for use in the ministerial environment and which potentially differ from ministry to ministry. The structuring means and functionalities provided by the CDR are driven by the requirements of the task-oriented and flexible cooperation activities within the user community. The CDR does not enforce a strict organizational model of the individual ministries, departments, etc. The cooperation structures encountered in the application domain do not rely on these organizational models but instead involve work groups spanning organizational bodies and set up for short- to medium-term activities. Also, the CDR is used as the unifying basis for more informal cooperations such as unplanned ad-hoc activities. These informal collaboration tasks would more often be hindered than simplified by adherence to too strict an organizational model.
186
Daniel A. Tietze, Ajit Bapat, Rolf Reinema
Document-Based Collaboration To give an impression of the functionality of the CDR and to provide an illustration of how the integrated system facilitates the cooperation processes, a short scenario is presented in the following. Suppose a user is member of a working group which is to come up with a document on the security of firewall technology for protecting the Governmental IP backbone. Assume that this user has been given the task to prepare an initial draft for this document. The cooperative scenario could, e.g., look like this: 1. In the CDR the user opens the repository that has been set up for the working group in question. With a double-click on the document the associated external application (here: a text processor) is started from within the CDR and the user can carry out his task of preparing a draft. 2. In the course of his work he might reach a point where a direct discussion with other members of the working group becomes necessary. From the list of group members the user selects his collaboration partner(s), indicates to the system that the document he is currently working on is to be subject of the collaboration and then simply invokes a „collaborate“ command from a pop-up menu within the CDR. The CDR then automatically performs all steps which are necessary to set up this document-based ad-hoc conference: a) An audio/video conference is set up between all collaboration partners. To enable a multipoint conference a corresponding conference is registered on the MCU. The MCU can then call all participants and establish the multipoint A/V conference. All data necessary for this technically complex step is available in the system (the telephone numbers, the types of the involved end points, etc.). After they accept the call, the video window is finally opened on each of the users’ desktops. b) The application sharing package that is needed to cooperatively edit the document is started for every participant in the conference. Here again, all technical information (e.g. connection numbers) is available in the system and is used to set up the sharing session without any manual interaction by the users. The text processor that is to be used cooperatively appears on the desktops of the conference participants. 3. Now that the working group has come together, the draft preparation can continue. The contents can be discussed over the audio/video connection and changes or extensions can be entered by any of the participants. 4. Users can easily input paper documents to the system by using the integrated scanner solution. The scanned document is directly imported into the group’s document repository. Documents can of course also be printed if wanted.
Document-Centric Groupware for Distributed Governmental Agencies
187
5. The participants can leave the conference at any time individually. All related connections are closed automatically and finally the whole conference is shut down. This example illustrates that working on documents in the repository as well as seamlessly switching from single user work to multilateral cooperative tasks requires only a minimum of user actions while the system can perform all complex „supporting actions“ automatically. In the following, some features of the CDR are presented in more detail. Initiation of Cooperative Sessions In order to cooperatively process a document from the CDR together with other users of the system, ideally all the user needs to indicate to the CDR desktop is which document he wishes to process together with which other users. Since the system is provided with all the necessary information about the users’ infrastructure, such as the phone numbers of the ISDN videoconferencing system available to the user, the supported bandwidths, etc., it is able to schedule and establish a collaboration session without further user input – apart from the obvious interaction steps of confirming the communication partners’ readiness to enter the collaboration session now or to postpone it to a later point in time. Since the documents which form the basis for the collaborative activity are already present in the CDR, distribution of these documents also becomes an easy task. Conference Scheduling A reservation system is available which generates all technical entries required by the MCU database and manages reservations of the required team rooms. A small part of the data needs to be input by the user (names of the participants, the date, time and duration of the conference etc.), while a major part of the information is stored in and retrieved from the CDR database (e.g. telephone numbers for the A/V connection, type of A/V hardware). Thus, an operatorless operation of the MCU is possible. Should no MCU resources be available for the chosen time slot, the user will be immediately informed that the conference is not possible at the chosen time. In addition to the MCU resources, any team rooms that are involved are checked for availability and are then reserved for the new conference. The newly created conference automatically appears in the conference list on the desktops of all conference participants, thus informing them about the new conference. When the time of the conference arrives, the MCU either automatically dials out to all participants or the CDR notifies the user about a conference being started. As soon as the user has confirmed his readiness the appropriate connections are established (again without requiring the users to remember and dial any numbers etc.). The initiation of ad-hoc conferences is achieved in the same way, simply by scheduling the start of the conference with the current date and time.
188
Daniel A. Tietze, Ajit Bapat, Rolf Reinema
Integration of External Applications Opening a document within a conference will indicate the user’s desire to present this document to the entire group of connected users or to edit it cooperatively with them. The CDR supports this activity by automatically launching not only the required application but also the application sharing component – if necessary, on all connected machines – and bringing the document’s application into the application sharing session ready for collaboration. This support relieves the users of a large part of the burden involved in setting up the conference connections, the required applications and the application sharing system. Should a document be editable or presentable by a truly collaborative software such as the DOLPHIN [7] system, which does not require the additional application sharing support, this integration is also available in the CDR. Again, since the CDR has access to all the necessary information about the collaboration situation, the external collaborative application can be invoked on all sites, receiving the required parameters for connection and setup from the CDR. Summary Using the integrated collaboration environment provided by the CDR in conjunction with the integrated communication and collaboration tools as presented above, the users can initiate and conduct their collaboration tasks in a manner which is appropriate to the problems they actually need to tackle. Indicating the collaboration partners and the document(s) on which to collaborate is sufficient for the system to set up the required communication channels, make the required reservations and start the necessary tools at the right point in time. The users are relieved of the effort of having to control (and learn) a number of separate applications and can concentrate on the actual work processes.
Related Work A number of research and development projects have taken on the task of supporting document-based collaboration between multiple distributed sites. This section compares representatives of these systems to the requirements listed in this paper and addressed by the CDR. The BSCW shared workspace system [8] is a Web-based groupware application which follows a collaboration paradigm similar to the CDR: users can create and structure repositories into which they can import documents which are then made available to all users who currently have access to the repository. The documents can be retrieved by the users, processed and at a later stage put back into the BSCW system for the other users. As a purely Web-based application it differs from the CDR in a number of points which were of importance in our application context (as shown in the requirements section). Firstly, BSCW is an asynchronous system which does not provide direct feedback and information about other user's actions (synchronous group awareness; R2.1). In order to observe changes within the repository, the Web page has to be manually reloaded by the user. Also, BSCW
Document-Centric Groupware for Distributed Governmental Agencies
189
provides no integration of synchronous multi-party A/V communication (R2.2 and R2.3). The CoopWWW [9] project aims at extending the BSCW system through the integration of A/V conferencing between the system's users. It currently does this by integrating a software A/V communications module, CUSeeMe [10]. The resulting system does not offer the high-quality A/V conferencing supplied by the CDR with the integrated ISDN conferencing system. Since CUSeeMe is a software solution and employs the Internet for data transport, the available bandwidths and thus the achievable frame rates and picture quality are much lower than the ones achieved by integrating ISDN-based multipoint conferencing as we have done in the CDR. An evaluation of user requirements has shown, though, that audio and video quality are of utmost importance for the acceptance and wide-spread use of the resulting technical solution (R2.2). Lotus Realtime Notes [11] is a combination of Intel ProShare and Lotus Notes. Notes, as a replicated distributed document database can be used to store documents which are accessible to a group of users. The use of Intel ProShare allows real-time communication and collaboration on these documents by use of the application sharing component. What this combination does not achieve, though, is the seamless integration of multipoint communication and multi-party distributed settings (R2.3). Also, this combination does not provide an integrated user interface with a familiar look-and-feel (Requirements R3.1, R3.2 and R3.3). The CDR goes one step further towards a complete integration by integrating scheduling, control and reservation of multiparty conferences. Furthermore, Realtime Notes does not provide synchronous group awareness to the system's users. Neither Realtime Notes nor BSCW/CoopWWW are adaptable to different standards-based collaboration and communication tools in the way in which the CDR is (R5.1). The TeamRooms system [12] also aims at supporting collaboration of distributed teams by providing persistent repositories for the collaboration documents. TeamRooms, though, does not integrate synchronous multimedia conferencing (R2.1, R2.2, R2.3) and uses specialized "applets" for cooperative document creation, thus has no provision for legacy applications and seamless integration into the collaboration processes (R1.1, R1.2).
Conclusions and Future Work This paper has presented the design and implementation of the CDR application, a system designed to support cooperative document management and document processing in a ministerial or governmental setting. It facilitates group cooperation and communication by integrating existing off-the-shelf components into a single comprehensive user interface which allows easy control of the various functionalities available in the different components, which were previously not integrated and therefore complex to employ in a productive manner. The adherence
190
Daniel A. Tietze, Ajit Bapat, Rolf Reinema
to the metaphor of document-based collaboration as a means of interacting with the system and performing the cooperative tasks has been implemented after an analysis of the collaborative tasks within the context of distributed governmental agencies. Even though the design and development of the CDR was strongly driven by its current application context of distributed governmental agencies, we believe the system is flexible and powerful enough to be usable in a wide range of application domains even beyond the public administration sector. Its approach of supporting the cooperation and communication tasks involved in joint document creation and document-based meeting support as well as its foundation on commercially available system components make it usable in a large number of distributed settings, e.g. in virtual organizations or distributed teams. The CDR has been installed in the user community and we are currently gathering feedback from the users which will directly influence the further development of the collaboration solution. Possible extensions of the system include enhanced document management functionalities, e.g. the support of different document versions within the repository as well as advanced document indexing and retrieval features. An extension of the architecture to include several distributed database servers might become necessary in order to better support the distributed setting and address scalability issues. Other aspects that will be examined in the ministerial application context is the use of secure audio/video conferencing, e.g. by using video encryption, as well as an advanced authorization concept for the team room setting to enable the use of the CDR application suite also in securitysensitive environments.
Acknowledgements The authors would like to thank the following people for the effort they put into the development of the software described in this paper: Marc Volz, Bernd Bussek, Oguzhan Karaduman, Fritz Lorenz, Sascha Lüdecke, Michael Pauly. We are also grateful to Jörg Haake for valuable comments about preliminary versions of this paper.
References [1]
Engel, A., Kaack, H., Kaiser, S.: Teamarbeitsräume zur Unterstützung verhandlungsorientierter Telekooperation; in: Rechnergestützte Kooperation in Verwaltungen und großen Unternehmen, Tagungsband zum Workshop der GI-Fachgruppe FG 551 und der GI-Fachbereiche FB6 und FB8 im Rahmen der Jahrestagung der Gesellschaft für Informatik; Aachen, 1997
[2]
ITU-T Recommendation H.320 (03/96) - Narrow-band visual telephone systems and terminal equipment; available electronically from http://www.itu.ch/itudoc/itu-t/rec/h/h320_23397.html; 1996
Document-Centric Groupware for Distributed Governmental Agencies
191
[3]
ITU-T Recommendation T.120 (07/96) - Data protocols for multimedia conferencing; available electronically from http://www.itu.int/itudoc/itut/rec/t/t120_35511.html; 1996
[4]
Microsoft ODBC 3.0 Software Development Kit and Programmer's Reference; Microsoft Press, 1997
[5]
Knopik, T.; Tietze, D.; Volz, M.; Paul, B.; Heite, R.; Speichermann, H.; Wittinger, C.: Towards a Collaborative Document Archive for Distributed Governmental Agencies; in: Lehner, Dustdar (eds): Telekooperation in Unternehmen; Dt. Univ.-Verlag, Wiesbaden, Gabler, 1997; pp. 65-78
[6]
Graham Hamilton, R. G. G. Cattell, Maydene Fisher: JDBC Database Access With Java: A Tutorial and Annotated Reference (Java Series); Addison-Wesley, 1997
[7]
Streitz, N.A., Geißler, J., Haake, J.M., and Hol, J. DOLPHIN: Integrated Meeting Support across LiveBoards, Local and Remote Desktop Environments. In Proceedings of the 1994 ACM Conference on Computer Supported Cooperative Work (CSCW '94), Chapel Hill, N.C., October 2226, 1994, pp. 345-358.
[8]
Appelt, W.; Busbach, U.: The BSCW System: A WWW-based Application to Support Cooperation of Distributed Groups. In: IEEE (ed.): Proc. of the Fifth Workshops on Enabling Technologies: Infrastructure for Collaborative Enterprises, June 19-21, 1996, Stanford, CA. Los Alamitos, CA: IEEE Computer Society Press, 1996, p. 304-309.
[9]
project description available at http://orgwis.gmd.de/projects/COOPWWW/
[10]
Information about CUSeeMe is available electronically http://bio444.beaumont.plattsburgh.edu/CUSeeMe.html.
[11]
Information available electronically from Lotus Realtime Notes Web site at http://www.lotus.com/home.nsf/welcome/realtime
[12]
Roseman, M., Greenberg S.: TeamRooms: Network Places for Collaboration; in: Proceedings of CSCW'96 (ACM Conference on Computer-Supported Cooperative Work); Cambridge, MA, USA; 1996
from
Specifying the Reuse Context of Scenario Method Chunks1 Colette Rolland1, Véronique Plihon2, Jolita Ralyté1 1
Université Paris1-Sorbonne, CRI, 90 rue de Tolbiac 75013 Paris, France {rolland, ralyte}@univ-paris1.fr 2 Université de Toulon et du Var, GECT, BP 132 83957 La Garde Cedex, France [email protected]
Abstract : There has been considerable recent interest in scenarios for accompanying many of the various activities occurring in the development life cycle of computer based systems. Besides the integration of scenarios in methods such as Objectory and software tools such as Rationale Rose has proven useful and successful. Consequently, there is a demand for adapting existing methods to support specific design activities using scenario based approaches. The view developed in this paper is that scenario based approaches should be looked upon as reusable components. Our concern is therefore twofold : first, to represent scenario based approaches in a modular way which eases their reusability and second, to specify the design context in which these approaches can be reused in order to facilitate their integration in existing methods. The paper concentrates on these two aspects, presents an implementation of our proposal using SGML to store available scenario based approaches in a multimedia hypertext document and illustrates the retrieval of components meeting the requirements of the user by the means of SgmlQL queries.
1. Introduction Scenario based approaches have proven useful in a large number of situations occurring in the system development life cycle. In the HCI community, scenarios have been proposed as detailed descriptions of a usage context so design decisions can be reasoned about [3] or as small examples of an existing product which are used to anchor discussion about different design theories [39]. In Software Engineering, use case approaches have been developed to derive the object oriented specification of a system from narrative descriptions of interactions with its users. In the Information Systems community scenarios have evolved to the concept of a rich picture which gives the social setting of a required system so arguments can be 1
This work is partly funded by the Basic Research Action CREWS (ESPRIT N°21.903). CREWS stands for Cooperative Requirements Engineering With Scenarios. B. Pernici and C. Thanos (Eds.): CAiSE’98, LNCS 1413, pp. 191-218, 1998. Copyright Springer-Verlag Berlin Heidelberg 1998
192
Colette Rolland et al.
developed about the impact of introducing technology, and the matching between user requirements and task support provided by the system [19]. Finally in Requirements Engineering, scenario scripts based approaches have been proposed to support the checking of dependencies between a requirements specification and the user/system environment in which it will have to function [25]. These examples demonstrate that a scenario based approach aims primarily at supporting some specific design activity. By essence these approaches are not standalone products but instead, they have vocation to be integrated in existing methods to support some specific steps of the design process with the advantage of increasing usability. As a specific scenario based approach provides support to a specific design activity, it might be possible to integrate it in various different methods dealing each with this particular design activity. Our view is that scenario based approaches should be looked upon as reusable components. This reuse perspective has been already illustrated, for example by the use case approach originally developed by Jacobson [14], [15], and then, integrated in a number of existing methods including the Fusion method [5], OMT [33], [34] and UML [1]. However reuse has been performed in an ‘ad hoc’ manner while there is a demand [16] for a more systematic way of understanding when, why and how, which kind of scenario has to be used. Thus, if we want to support the reuse of the large corpus of available scenario based approaches we shall solve the problem of characterising the context in which they can be reused. In this paper we are concerned by these two issues : (a) to represent scenario based approaches as reusable components and (b) to specify the context of use of available scenario based approaches in order to facilitate their reuse in different methods to support the design activities they are dedicated to. Our proposal is based on an analogy with object oriented reuse and comprises two aspects: 1. To define a scenario based approach as a collection of methods fragments that we call scenario method chunks (scenario chunks for short) and to make them available in a scenario method base. 2. To characterise the context of use of scenario chunks in chunks descriptors and to store them in the scenario base together with the chunks themselves. This shall ease the retrieval of scenario chunks meeting the requirements of the method base user. Our view is therefore to organise the scenario method base at two levels, the method knowledge level and the method meta- knowledge level and to tightly couple scenario chunks with the meta-knowledge describing their context of use. The method level tells us how to apply a specific scenario chunk whereas the meta-level provides knowledge about the conditions under which the scenario chunk is applicable. Our notion of chunk descriptor is close to the view of [6] and very similar to the one of faceted classification schema [27] developed in the context of software reuse.
Specifying the Reuse Context of Scenario Method Chunks
193
We have implemented the proposed approach using SGML (Standard Generalised Markup Language). The two knowledge levels of the method base are parts of the same SGML document in order to facilitate their joint manipulation. Our motivation for using SGML has been on the one hand, its ability to represent hyper-text documents and on the other hand, the availability of sgmlQL which is an SQL like language tailored to query SGML documents. This provides the required facilities to query the scenario method base and retrieve the scenario chunks which match specific reuse conditions. In the rest of the paper, we develop in detail the scenario method knowledge and the scenario meta-method knowledge as well as their implementation. Section 2 deals with the former, presents the notion of scenario chunk, illustrates the different levels of granularity of chunks and exemplifies them by describing several existing scenario based approaches. Section 3 deals with the meta-knowledge representation, defines and exemplifies the notion of chunk descriptor. Section 4 covers the implementation of the scenario method base in SGML. In section 5 we illustrate through examples of queries in sgmlQL how scenario chunks can be retrieved from the method base. Finally we draw some conclusions in section 6.
2. Scenario Method Knowledge Level We adopted a modular approach to represent the scenario method knowledge, in the method base, in the form of scenario method chunks, scenario chunks for short. A scenario chunk may represent an entire approach such as the Jacobson’s use case approach or part of it, for example the chunk to define abstract use cases. This eases the reusability of chunks and their integration in methods. A chunk tightly couples a product part and a process part. In the product part, the product to be delivered by a scenario chunk is captured whereas in the process part, the guidelines allowing to produce the product are given. As we are interested in scenario chunks, at least one of the product parts involved in a chunk must be of the scenario type. The guidelines to define the use case model proposed in the OOSE methodology [13], to capture and use scenario scripts [25], to construct interaction diagrams [35], or abstract usage views [28] are examples of such scenario chunks. 2.1 The Notion of Scenario Chunk Our definition of a scenario chunk is based on the process view of the NATURE process modelling formalism [30], [24] and consistent with the notion of ‘step’ in [37] . According to this view a process can be seen (Fig. 1) as a black box which transforms an initial situation into a result which is the target of the intention of the process. The situation represents the part of the product undergoing the process and the intention reflects the goal to be achieved in this situation. The target of the intention is the result produced by the process execution. As the target is embedded in the intention, this leads to the characterisation of a process by a couple <situation, intention> which is called context (see the example in Fig. 1).
194
Colette Rolland et al. Intention ex : Define (Use Case model) Situation (input) ex : Problem description
Scenario Chunk
Target (output) ex : Use Case model
Fig. 1. The behavioural view of a scenario chunk
Following this view, a scenario chunk has two parts (Fig. 2)2 : its interface which is the couple <situation, intention> and a body. We chose these designations by analogy with object descriptions in object oriented approaches. The interface is the visible part of the chunk. It tells us in which situation and for which intention the chunk is applicable. The body explains how to proceed to fulfil the intention in that particular situation. The body provides guidelines to guide the process and relates the process to the product parts involved. For example, the interface of the scenario chunk representing the Jacobson’s use case approach is the context <(Problem Statement), Define Use Case Model > where the situation is represented by a document called problem statement and the intention, to define use case model, is based on this problem statement. The target of the intention, use case model defines the result to be produced by the application of the chunk. The body provides the guidelines to achieve the intention of defining a use case model out of problem statements (see Fig. 4 that we will explain later on). Name Graphical representation
has Scenario Chunk
Situation Interface Intention
Informal description has Example Type
has target
Body
refers to
is based on Guideline
Product
Product Part
Fig. 2. The scenario chunk structure
Additionally, as shown in Fig. 2, each scenario chunk in the method base has a unique name. It may be represented graphically and /or described informally in natural language. It is also possible to provide some examples of anterior application of this scenario chunk. A scenario chunk is classified either into formal or informal. In the former case the guidelines are formally defined whereas they are informal textual recommendations in the latter. 2
The structure description is based on E/R like notation.
Specifying the Reuse Context of Scenario Method Chunks
195
As illustrated in the example above, the intention contains the reference to the target. In fact the structure of an intention (Fig. 3) is more complex than a simple verb. The proposed formalisation of the notion of intention permits a fine grain characterisation of the chunk which was felt necessary to support efficient retrieval of scenario chunks. Intention Verb
Manner
Target is a Object
#
is a Result
Fig. 3. The intention structure
The intention is decomposed [26] into a verb, a target (a product part) the verb is acting on and a manner. Depending on the role played by the product part for the verb, we make the distinction between objects and results. An Object exists in the situation of the corresponding context whereas a Result is produced by the achievement of the intention. “Refine use case”, is an example of intention in which the target “use case” is an object because it already exists in the situation whereas “Identify actor” is an example where the target “actor” is a result. It is developed during the execution of this intention. The precise notation of these intentions is as follows: <(Use Case), Refine (Use Case)Obj>, <(Problem Statement), Identify (Actor)Res>. In addition to the verb and the target, the intention may have a manner which specifies in which way the intention is satisfied. A manner is a strategy or an approach used to fulfil the intention. «One-shot refinement» or «stepwise strategy» are examples of manners. The proposed definition of a scenario chunk is applicable to any method chunk. The distinction between a scenario chunk from any other method chunk is due to the nature of the product parts. In the latter the product can be of any type whereas in the former either the situation or the target of the intention must refer to a product part of the scenario type. For example, the two chunks with the following interfaces <(Problem Statement), Define (Use Case Model)Res> and <(Use Case Model), Interpret <(Use Case Model)Obj> are scenario chunks. Both manipulate scenarios which are called use cases, the former having the use case model as the target of the intention to Define Use Case Model whereas the use case model is the object of the Interpret intention. The application of the NATURE contextual approach to the representation of method knowledge has the important effect of making chunks modular. A chunk prescribes the way to proceed in a situation to fulfil an intention. A scenario chunk can be qualified as cohesive because it tells us the situation in which it is relevant and the
196
Colette Rolland et al.
intention that can be fulfilled in this situation. A chunk is loosely coupled to other chunks because it can be used in the appropriate situation (created as the result of another module) to satisfy the intention. Thus, the linear arrangement of method modules is replaced by a more dynamic one. Finally, the hooks to combine a scenario chunk with another chunk (whichever is its type) are parts of its interface : the situation and the target. Two chunks can be assembled if the situation of one of them is compatible with the target of the other. 2.2 The Body of a Scenario Chunk The interface of a scenario chunk characterises the conditions of its applicability whereas its body details how to apply it. The interface plays a key role for retrieving a scenario chunk out of the method base while the body is used when applying the method in which the chunk has be integrated. Our approach relies upon the interface structure presented above but does not imply a particular way of describing the chunk body. In the sequel, we illustrate partially the solution we chose for the implemented method base. Plan Context c4
<(Pb.St.); Identify Actor> c2
cc7
<(Pb.St., Primary Actor); Define Use Case> c5
c1
<({Use Case}); Refine Use Case> c6
c3
c1: The mode is width first and all the actors have not been identified yet c2: ... Refinement Link c3: a1 or a2 and a3... a1: there is a non implemented optional <({Use Case}); <(Problem Statement); <(Problem Statement, Primary Actor); functionality Refine Use Case> Identify Actor >* Define Use Case>* a2: ... c3 c1 c2 <(Use Case); <(Problem Statement); Complete Use Case with <({Use Case}); Identify Primary Actor >* Use Case Extension > Generalise {Use Case} <(Problem Statement); into Abstract Use Case> Executable Identify Secondary Actor >* <(Abstract Use Case); Context Action Specialise Abstract Use Case Link into Concrete Use Case> Composition Link
<(Problem Statement); Define Use Case Model>
Choice Context
... ...
... ...
... ...
Action: create actor
... ... Fig. 4. Excerpt of a scenario chunk [13]
We follow the NATURE approach and define the chunk body as a hierarchy of contexts called a tree. As illustrated in Fig. 4 contexts relate one to the other through three types of links : refinement links which permit the refinement of a large-grained context into finer ones, composition links which allow to decompose a context into component contexts and action links which relate the contexts to the actions which directly perform transformations on the product. Each type of link has a
Specifying the Reuse Context of Scenario Method Chunks
197
corresponding type of context, namely executable, choice and plan contexts. A detailed description of contexts can be found in [31]. Let us briefly illustrate them through the example of tree presented in Fig. 4. A choice context offers choices supported by arguments. The context <({Use Case}), Refine Use Case> in Fig. 4 introduces three alternatives to a Use Case refinement : (1) to generalise a set of use cases into an abstract use case <({Use Case}), Generalise {Use Case} into Abstract Use Case>, (2) to specialise an abstract use case into concrete use cases <(Abstract Use Case), Specialise Abstract Use Case into Concrete Use Case> or (3) to extend a use case with a use case extension <(Use Case), Complete Use Case with Use Case Extension>. A plan context corresponds to an intention which requires further decomposition. The plan context <(Problem Statement), Define Use Case Model> Fig. 4. is composed of three contexts, namely <(Problem Statement), Identify Actor>, <(Problem Statement, Actor), Define Use Case> and <({Use Case}), Refine Use Case>. This means that, while constructing a use case model, the requirements engineer has first to identify the actors, then to construct use cases, and finally, to refine the use cases. The component contexts of a plan context can be organised within graphs in a sequential, iterative or parallel manner (see the top bubble in Fig. 4). An executable context corresponds to an intention which is directly applicable through actions which transform the product under development. Performing the action changes the product and may thus generate new situations. In Fig. 4 the context <(Problem Statement), Identify a Primary Actor> is an executable context. The intention to "Identify a Primary Actor" is immediately applicable through the performance of an action which creates a new "primary actor" in the use case model under development. Notice that when the chunk is informal, its body is reduced to a textual explanation on how to perform the action (see Fig. 5). Contexts can therefore be related through refinement and composition links to form a tree. A tree is considered complete when all its leaves are executable contexts. The scenario chunk in Fig. 4 is a tree in which the root context <(Problem Statement), Define Use Case Model> is decomposed and refined throughout the hierarchy of contexts to the set of executable contexts. Thus, the global intention to define use case model is transformed into a set of actions which transform the product to develop the use case model. For the sake of clarity, the tree is only partially represented in Fig. 4. Finally, our concrete experience in constructing the SGML scenario knowledge base has raised difficulties due to the lack of formal descriptions of either the process or the product models of the scenario approaches. Moreover, approaches rarely include the process dimension. To cope with this situation, we classify scenario chunks into formal and informal (see Fig. 2) and propose to associate to informal chunks textual explanations on how to fulfil the intention of this type of scenario. For example, in Fig. 5 we present a scenario chunk defined according to Kyng’s approach [19]. This chunk proposes to describe work situations in which the users identify some
198
Colette Rolland et al.
problems and bottlenecks. The author does not detail how to work with these scenarios, he only explains what type of information these scenarios have to contain.
Intention : Verb : Capture Result : Work Situation Description
Situation : Product Parts : {Key Insight}, {Summary} Description : Key Insights and Summaries are reflecting the initial study of the work places.
Informal Description : The aim of the chunk is to support the description of work situations to identify problems and bottlenecks in these situations.
< ({Key Insight}, {Summary}), Capture Work Situation Description > Informal Body : «Work Situation Descriptions are descriptions of relevant, existing situations within the users work-place. The users find that these situations are important parts of their work and that currently they constitute a bottleneck, are error prone, or for other reasons need to be changed...» [Kyng 95]
Fig. 5. Scenario chunk from Kyng's approach
2.3 Scenario Chunk Granularity Fig. 6 sums up the three different levels of granularity of scenario chunks: contexts, hierarchies of contexts called trees which may be parts of a scenario approach. Each of these chunks can be considered either independently or as part of an overall scenario approach. A scenario based approach is viewed itself as a chunk (is-a link in Fig. 6). Indeed, both the approach itself and its component chunks are reusable. Typically, a scenario approach contains guidelines for the creation, the transformation and the refinement of scenarios into more conceptual products. For example, in the OOSE methodology [13], we identified two scenario chunks, one to construct the use case model and a second one to construct the analysis model out of the use case model. The composition of these two chunks corresponds to a scenario approach which is also proposed in the method base as another chunk.
composed of 1,1 1,N Scenario Scenario is a Scenario Scenario Approach Chunk Approach Chunk is a
#
Tree 1,N
is a Context 1,N
composed of Fig. 6. Structure of the scenario knowledge in the scenario method base
Specifying the Reuse Context of Scenario Method Chunks
199
3. Scenario Method Meta-Knowledge Level The scenario method knowledge is about descriptions of available scenario method chunks. The scenario method meta-knowledge we are dealing with in this section aims at specifying the context in which method knowledge can be (re)used. Assuming that the scenario base has been constructed, the question addressed now is « how to ease the retrieval of scenario chunks meeting the requirements of a method engineer who wants to extend an existing method with scenario features?». This raises the need for understanding when, why and how a specific scenario chunk can be reused i.e. to specify the context of its use. Our literature survey [32] as well as the industrial visits performed within the companies of the CREWS steering committee [16] have shown that this knowledge is not available. Both have also demonstrated that there is an explicit need for making this knowledge available. Our view is that the knowledge about the context of use of scenario chunks shall be formalised and stored in the scenario method base with the scenario chunks themselves. We call this knowledge method meta-knowledge as it provides information characterising the use of scenario method knowledge. The scenario method base is therefore organised at two levels, the method meta-knowledge level and the method knowledge level. In the process of reusing scenario chunks, these two levels serve in separate steps. The method meta-knowledge supports the retrieval step whereas the knowledge is the material effectively reused and integrated in the existing method. In this section we are concerned with the meta-knowledge representation. We shall illustrate the use of this meta-knowledge in section 4 through sgmlQL queries acting on the implemented method base. We use the notion of descriptor [De Antonnellis91] as a means to describe scenario chunks. A descriptor plays for a scenario chunk the same role as a meta-class does for a class. Our concept of descriptor is similar to the one of faceted classification schema [27] developed in the context of software reuse. We extend the contextual view used to describe the chunk interface to structure the meta-knowledge in the descriptor. Indeed, we view the retrieval process as being contextual : a user of the method base is faced to reuse situations at which he/she looks with some intention in mind. Therefore, the descriptor seeks to capture in which situation a scenario chunk can be reused to fulfil which intention. If we remember that scenario based approaches primarily aim at supporting specific design activities in different ways, the descriptor situation shall refer to this design activity whereas the intention expresses a design goal related to this activity. As an example, the descriptor of the Jacobson’s chunk described in section 2 shall refer to ‘analysis’ as a design activity supported by the chunk and ‘capture user/system interactions’ as the intention within this activity which can be supported by the use case approach provided by the chunk. Then, our descriptor is contextual as it captures situational and intentional knowledge defining the context of reuse a scenario method chunk.
200
Colette Rolland et al.
Fig. 7 gives an overview of the knowledge captured in the method base for every chunk. The chunk body is actually the fragment of method to deal with a specific type of scenario whereas the chunk interface describes its conditions of applicability, the situation required as input of the chunk, and the intention the chunk helps to fulfil. These two aspects constitute the scenario method knowledge whereas the meta-knowledge is captured in the scenario descriptor. The descriptor expresses the reusability conditions of the chunk by characterising the design activity in which it can be reused (the situation part) and the design intention that can be supported by the scenario chunk (the intention part). It describes the broad picture in which the scenario approach captured in the chunk can take place. In the sequel, we develop the chunk descriptor in detail. Method meta-knowledge level
3.1 The descriptor structure Fig. 8 depicts the descriptor structure. A chunk descriptor has a situation part and an intention part that we consider in turn. Scenario Scenario Chunk Chunk Application Domain
The Situation Part of a Chunk Descriptor The situation part of a descriptor comprises two aspects (Fig. 8): the application domain and the design activity in which the scenario chunk is relevant. For instance,
Specifying the Reuse Context of Scenario Method Chunks
201
considering the Jacobson’s chunk (Fig. 4) which describes how to proceed for defining a use case model, the domain of the descriptor is Object Oriented Applications and its design activity is Analysis. This means that this chunk can be reused in Object Oriented Application for facilitating the Analysis step. While populating the scenario method base, we have identified a list of application domains in which scenarios are used. The current list of domains in the implemented scenario base is the following: • Usability Engineering • OO applications • Requirements Engineering • HCI (Human Computer Interfaces) • Workflow applications • Critical systems • Information systems • Socio-technical applications HCI (see [3] for a survey) and OO applications [4], [9], [13], [20], [28], [29], [36], [33], [38], [21] are the two domains where scenarios are nowadays extensively used. Similarly we have identified a list of design activities, (similar to the one proposed in [3], Table 1) each of which is supported by at least one scenario chunk. Design Activity Analysis Envisionment Requirement Elicitation Design Rationale Validation Software Design Software Testing Team Work Building Communication Documentation / Training
Table 1 : Design activities covered by different scenario chunks
The Intention Part of a Chunk Descriptor The chunk descriptor intention expresses how the scenario approach encapsulated in the chunk participates to the achievement of the design activity. For example, the intention of the descriptor of the Jacobson’s chunk presented in Fig. 4 is ‘capture user/system interactions’ as the chunk provides a scenario based approach supporting the capture of the interactions between the future system and its users. The descriptor intention is an expression of the role that a scenario approach can play in a particular design activity. We found in our survey of both literature and practice a large panel of roles, all being informally expressed and therefore difficult to classify and
202
Colette Rolland et al.
organise to support the retrieval process in the method base. In the following list we give some examples of our findings: • • • • • • •
Supporting the analysis of the users workplace and work situations Expressing how a system being designed should look like and behave Facilitating the discovery of user needs Helping evaluating possibilities for usability and functionality Supporting the identification of central problem domain objects Helping to develop cohesion in the team Facilitating the communication on the systems problems between users and designers • Helping to test whether the system satisfies or not all the user’s requirements • Supporting user training • Bridging the gap between the system presented as an artefact and the tasks the users want to accomplish using it Instead of using role names as described in the literature, we use a more formal description of intentions based on [26] leading to the intention structure presented in Fig. 8. This structure is compatible with the one used for the chunk interface. It is extended in order to link the intention of the chunk and the intention of its descriptor. The intention in the chunk descriptor is specified by the intention verb, the target of by this intention and the manner to satisfy this intention (Fig. 8). Let us detail these various elements of the intention structure in turn. Similarly to the target in the scenario chunk interface (see Fig. 3), the target of the descriptor is specified into object or result depending on the role played by the target for the verb. These roles have been explained in section 2. Moreover, in the chunk descriptor intention we make the distinction between non-scenario based target and scenario based target (see is a links in Fig. 8) • Non-scenario based targets represent product parts other than scenarios. Functional system requirements, non functional system requirements, object model, alternative design option, etc. are examples of non-scenario based targets. • Scenario based targets represent product parts of the scenario type. Use case, scenario script, episode, work situation description or use scenario are examples of scenario based targets. In order to ease the retrieval process, there is a need for characterising scenario targets with enough details to differentiate one from the other. Our characterisation is based on the framework defined in the CREWS project [32]. Properties such as the scenario formality, the level of abstraction, the nature of interactions, the covered requirements or the scenario life cycle are proved necessary to select more precisely the adequate scenario chunks. There are eleven properties which are surveyed in appendix 1. The chunk descriptor intention is completed by a manner which is a complex manner by opposition to a simple manner as used in the scenario chunk interface. A complex manner is recursively described as an intention. The intention to Capture user/system interactions by defining a use case model with Jacobson’s refinement
Specifying the Reuse Context of Scenario Method Chunks
203
strategy is an example of descriptor intention using a complex manner. The manner (by defining a use case model with the Jacobson’s refinement strategy) is recursively defined as an intention (defining) with a result ( use case model) and a manner (with the Jacobson’s refinement strategy). The intention is denoted as follows: Capture (User/System Interactions)Res (by Defining (Use Case Model)Res (with Jacobson’s Refinement Strategy)Man )Man. The descriptor intention always refers to a complex manner. This allows us to link the scenario chunk intention to the descriptor intention. This is modelled in figure 8 by an is a link from the manner to the chunk interface intention. In the example above, the intention to define use case model with Jacobson’s refinement strategy is the intention of the Jacobson’s chunk presented in Fig. 4. It is embedded in the descriptor intention as the manner of the intention to Capture user/system interactions. The scenario approach captured in a given scenario chunk is formally defined as the manner to achieve a design intention. Both the design intention and the manner to achieve it are expressed in the descriptor intention. 3.2 Examples In Fig. 9, we present the example of the descriptor corresponding to the Define Use Case Model scenario chunk depicted in Fig. 4.
Intention : Verb : Capture Result, Non-Scenario Based: User Interactions Complex Manner : by Defining Use Case Model...
<(OO Design, Analysis), Capture ( User Interactions )Res ( by Defining [ Use Case Model FormDescriptionMedium = Text FormDescriptionNotations = Informal Complex Manner is an Intention : ... Verb : Define Result, Scenario Based : Use Case ContentsCoverage = Functional Model ContentsContext = System Interaction Properties : Form... ... Contents.... Life CycleLife span = Persistent Atomic Manner : by Jacobson’s Purpose = Descriptive ] Res Refinement Strategy (by Jacobson's Refinement Strategy ) Man ) Man >
Fig. 9. Example of the Jacobson’s chunk descriptor
This descriptor tells us that the corresponding scenario chunk is useful in the domain of object oriented applications for supporting the analysis activity. The intention in this situation is to capture the interactions between the user and the system. Moreover, the intention is clarified by the manner to fulfil it. This manner defines precisely the way the scenario chunk will help fulfilling the intention of user/system
204
Colette Rolland et al.
interaction capture: it is by defining a use case model in the Jacobson’s style. Because the target of this intention (define use case model) is a scenario based one, the descriptor specifies its discriminant properties, namely the FormDescriptionMedium (textual), the FormDescriptionNotation (informal), the ContentsCoverage (Functional), the ContentsContext (user/system interactions), the LifeCycleSpan (persistent) and the Purpose (descriptive). As another example of chunk descriptor, Fig. 10 presents the descriptor associated to the informal Kyng’s scenario based approach whose chunk was sketched in Fig. 5.
Intention : Verb : Discover Result, Non-Scenario Based: Problematic Work Situation Complex Manner : by Capturing Work situation...
<(Socio-technical Application, Analysis), Discover ( Problematic Work Situation ) Res (by Capturing [ Work Situation Description FormDescriptionMedium = Text FormDescriptionNotation = Informal ... Complex Manner is an Intention : ContentsCoverage = Functional, Verb : Capture Non-Functional, Intentional Result, Scenario Based :Work Situation ContentsContext = Organisational Context Description ... Form... Properties : LifeCycleSpan = Persistent Contents.... Purpose = Descriptive ] Res Atomic Manner : in Participative (in Participative Workshop ) Man ) Workshop Man>
Fig. 10. Descriptor of the Kyng’s chunk
4. Using SGML to Implement and Query the Scenario Method Base SGML (Standard Generalised Markup Language) [10] is an international standard language to describe a document using a set of mark ups defined in a grammar. SGML documents are structured as trees. We found the language adequate for representing our scenario method base. Besides SgmlQL [22] is available to query an SGML base of documents. We sketch in this section the SGML structure of our implemented scenario method base and will sketch the query process in the next section. 4.1 Overview of the SGML Structure of the Scenario Method Base The structure of the scenario method base is described in a DTD (Data Type Definition). The whole DTD is a document type. It is composed of elements which are characterised by a mark up identifier, constraints on the existence of the opening and closing tags and the definition of their structure, i.e. the component elements.
Specifying the Reuse Context of Scenario Method Chunks
205
An element can recursively be composed of other elements. Based on this principle, the scenario method base is represented as a document type named METHODBASE (see Fig. 11) which is composed of the element named DESCRIPTIONS. This element consists in a set of descriptions which are themselves called DESCRIPTION. Thus, the structure of the element DESCRIPTIONS is represented by DESCRIPTION* where the «*» means many.
- - -
Name CHUNK* Graphical_ Representation BODY Informal_ Example INTERFACE Description GUIDELINE PRODUCT
CONTEXT_ Role Type INTENTION Product CONTEXT_INTENTION Scenario_Characteristic Name LINK* Graphical_ INFORMAL_ Representation CONTEXT_SITUATION SIMPLE_ RECOMMEN- Informal_ VERB TARGET MANNER Description DATION PRODUCT_PART* SITUATION_ Example DESCRIPTION? ACTION COMPOSITION REFINEMENT _LINK _LINK _LINK ACTION
CONTEXT
CONTEXT
ARGUMENT?
Fig. 12.Overview of the SGML structure of the scenario method base
This way of modelling is recursively applied to integrate all the elements composing the document type. Thus, the DESCRIPTION element is characterised by a metaknowledge level (META_KNOWLEDGE_LEVEL) and a knowledge level (KNOWLEDGE_LEVEL) which are, as presented in the previous sections, respectively composed of a descriptor (DESCRIPTOR), and of either a chunk or an
206
Colette Rolland et al.
approach denoted by (CHUNK | APPROACH). The resulting structure of the METHODBASE document type is the tree presented in Fig. 12. It is possible to attach attributes to elements to characterise them. For example, the attribute TYPE attached to the element CHUNK characterises its type (FORMAL, INFORMAL). An attribute has a name, a type (enumerated or not) and may be optional (#REQUIRED, #IMPLIED). Underneath is the Sgml description of the mandatory attribute TYPE of CHUNK.
CHUNK TYPE
(FORMAL | INFORMAL)
#REQUIRED>
The overall description of the document type defined for the scenario base is provided in the appendix 2. In the following section, we illustrate the Sgml document contents with the Jacobson’s chunk and its associated descriptor. 4.2 Examples of Sgml Chunks and Chunk Descriptors Let us start with chunk descriptors. As explained in section 3, a chunk descriptor has a situation part and an intention part. According to our approach, the situation part (see DESCRIPTOR_SITUATION in Fig. 13) is characterised by an application domain (APPLICATION_DOMAIN) and by a design activity (DESIGN_ACTIVITY) which explain respectively, the area and the design activity in which the chunk can be reused. In our example, the area is Object Oriented Applications and the activity is Analysis. <APPLICATION_DOMAIN>Object oriented applications analysis Fig. 13. Example of descriptor situation
The intention part of the descriptor (DESCRIPTOR_INTENTION) is illustrated in Fig. 14. It is composed of: • a verb (VERB), to Capture in our example, • a target (TARGET) which can either play the role of a result or of an object. This information is denoted in the attribute role of the target by the values « result » and « object ». In the intention Capture user/system Interactions, the target User Interactions is considered as a result. Moreover, the target is either a scenario-based product (like Use Case Model in Fig. 14) or not (like User/system Interaction in Fig. 14). In the former case the target has a number of characterising properties represented as attributes (FormDescriptionMedium, FormDescriptionNotation, ...) attached to the TARGET element. • a manner (COMPLEX_MANNER) As presented in section 3, the manner of the intention in the chunk descriptor is a complex manner (COMPLEX_MANNER) whereas the one of the intention in the
Specifying the Reuse Context of Scenario Method Chunks
207
chunk interface is a simple manner (SIMPLE_MANNER). A simple manner is represented by a string (#PCDATA) whereas the complex one has the structure of an intention. In Fig. 14, «Jacobson’s Refinement Strategy» is a simple manner whereas «by Defining Use Case Model by Jacobson's Refinement Strategy» is a complex one. Captureuser interactionsDefiningUse Case Model <SIMPLE_MANNER> by Jacobson’s Refinement Strategy Fig. 14. Example of Sgml descriptor intention
Now that we are aware of the Sgml description of the meta-knowledge level of the scenario method base, let’s concentrate on the knowledge level. The knowledge level (KNOWLEDGE_LEVEL) (see Fig. 12) is represented in the Sgml structure either by a chunk element (CHUNK) or by an approach (APPROACH) which is composed of chunks (CHUNKS*). As illustrated in Fig. 15, the chunk (CHUNK) element contains two parts : an interface (INTERFACE) and a body (BODY). It has general characteristics, namely a name, a type (formal, informal), an informal description and a reference to a graphical representation. Problem Statement
208
Colette Rolland et al.
<SITUATION_DESCRIPTION>The problem statement is an initial textual and informal description of the expectations about the future system. It contains some requirements and constraints resulting from interviews with the end users DefineUse Case Model <SIMPLE_MANNER>by Jacobson’s Refinement Strategy Fig. 15. Example of Sgml description of a chunk
The interface is composed of two parts : a situation (CHUNK_SITUATION) and an intention (CHUNK_INTENTION). The situation of the chunk interface (CHUNK_SITUATION) is composed of two elements: • one or several product parts referenced by PRODUCT_PART* in the Sgml tree, and • a description (SITUATION_DESCRIPTION) which is optional. All these elements are strings (#PCDATA).
Specifying the Reuse Context of Scenario Method Chunks
209
The intention of the chunk (CHUNK_INTENTION) is composed of a verb (VERB), a target (TARGET) and a simple manner (SIMPLE_MANNER). This is exemplified in Fig. 15. Following our definitions in section 2, the body is composed of two parts, the product and the guideline. • The product (PRODUCT) is characterised by a name, an informal description, an example of instantiation of the product and a reference to a graphical representation which is a picture stored in the Sgml document. This graphical representation is referenced in Fig. 15 by JacobProd.gif and is presented in Fig. 16. composed-of
Use Case composed-of Model 1,N
1,N 1,1
Actor Topic Secondary Actor
Primary Actor
executes
1,N
1,1 1,1
Description 0,N
Use Case extends
0,N
1,N
1,1
supports
Extension Use Case Concrete Use Case
Abstract Use Case
0,N
1,N
uses
Basic Use Case 0,N
Alternative Use Case 1,1
has
Fig. 16. JacobProd.gif
• The guideline (GUIDELINE) can be either represented by an informal description (INFORMAL_RECOMMENDATION) or by a set of links (LINK*) depending on whether the chunk is informal or not. In the case of a formal chunk, the guideline has the form of either a context or a tree of contexts. It is represented in the Sgml structure by the set of links, connecting the contexts one with the others in the tree. Depending on the type of its source context, a link can be either a composition, a refinement or an action link (Fig. 15). The tree structure can be visualised through the graphical representation (GRAPHICAL_REPRESENTATION) element of the structure. This graphical representation is referenced in Fig. 15 by JacobProc1.gif and is presented in Fig. 17.
210
Colette Rolland et al. <(Problem Statement), Define (Use Case Model) Res (by Jacobson's refinement strategy) Man >
<(Abstract Use Case), <({Use Case}), Generalise ({Use Case}) Obj Specialise (Abstract Use Case) Obj into (Concrete Use Case) Res> into (Abstract Use Case) Res> <(Use Case), Complete (Use Case) Obj with(Use Case Extension ) Res > < (Pb. St., Actor), < (Pb. St., Actor, UCTopic), Identify (Use Case Topic) Res >* Construct (Use Case) Res>
< (Pb. St.), Identify (Secondary Actor ) Res>* < (Pb. St.), Identify (Primary Actor ) Res>*
< (Pb. St., Use Case), < (Pb. St., Actor, UCTopic), Identify (Extension Use Case Topic) Res> Construct (Concrete Use Case) Res> < (Pb. St., Actor, {UCTopic}), <(Pb. St. Extension UCTopic), Construct (Abstract Use Case) Res > Elaborate (Extension Use Case Description) Res> <(Pb. St., Actor, {UCTopic}), <(Pb. St., Abstract UCTopic), Identify (Abstract Use Case Topic) Res> Elaborate (Abstract Use Case Description) Res> < (Pb. St., Actor, UCTopic), <(Pb. St., Basic Use Case), Elaborate (Basic Use Case Description) Res> Elaborate (Alternative Use Case Description) Res>
Fig. 17. JacobProc1.gif
5. Examples of SgmlQL Queries This section illustrates the way SgmlQL queries can support the process of reusing scenario chunks. This illustration is based on a reuse scenario that can be described in the following way. Let’s assume that Mr Bean,. a requirements engineer is in charge of a project for developing an object oriented application. In the early phase of the project the project used the Enterprise Modelling approach [2] to model the rich picture of the organisational context in which the system will operate. The result of this step is a set of high level goals. Mr Bean is now faced to the operationalisation of these goals i.e, the specification of the system functions which fulfil these goals. The project constraint is that the system functional specification should be done in an object oriented manner. Mr Bean’s belief is that the project should benefit from using a scenario based approach. The argument in favour of such an approach is twofold : (1) the system is too large and complex to be tackled in a global way and, (2) the domain experts are not familiar with semi-formal or formal notations and prefer to communicate with requirements engineers in an informal manner. Thus, we propose to help Mr Bean by querying the Sgml scenario method base to retrieve chunks that match his requirements i.e. chunks which help bridging the gap between goal models and object oriented specifications. We first propose to extract from the method base all the chunks whose situation is based on goals. The query is formulated as follows:
Specifying the Reuse Context of Scenario Method Chunks
211
Q1: Select the chunks having goals as product parts of their situation. select text($a->NAME) from $d in every DESCRIPTION within $myfile3, $a in every APPROACH within $d, $pp in first PRODUCTPART within $a where text($pp) match «goals»; The answer to this query proposes two chunks, the: • Holbrook’s, and • Cockburn’s ones. Mr Bean is not familiar with the Cockburn’s approach, but he heard of the Holbrook’s one and wish to explore further the possibilities offered by this approach. As the constraint is to produce an object oriented specification, the question is to identify which is the output of the Holbrook’s chunk. Q2 gives the answer to this question. Q2: Select the target generated by the « Holbrook » chunk. select text(first TARGET within $cm) from $d in every DESCRIPTION within $myfile, $descro in every DESCRIPTOR within $d, $cm in every COMPLEXMANNER within $descro, $a in every APPROACH within $d where text($a->name) match « Holbrook » ; The target is: Use Scenario An access to the PRODUCT part of the chunk in the Sgml document (to the HolbProd.gif in particular) convinces Mr Bean that unfortunately, the output of the Hoolbrook’s chunk is not an object oriented specification. Thus, we suggest to search for another scenario based chunk that supports the transformation of input scenarios into object-oriented models. This can be done with query Q3 , presented below. Q3: Select chunks which are using scenario or scenario-based product as input and generate an analysis model as output. select text($c->NAME) from $d in every DESCRIPTION within $myfile, $descro in every DESCRIPTOR within $d, $c in every CHUNK within $d, $pp in first PRODUCTPART within $c where ((text($pp) match «scenario» ) or (text($pp) match «use-case» )) and (text(first TARGET within $descro) match «analysis model»); This query results in the selection of the chunk named « Define analysis model ». Mr Bean is happy and decides to explore further on the solution consisting of combining
3
A preliminary query (global $myfile = file « MethodBase.sgml ») should be typed in order to indicates which sgml document (here MethodBase.sgml ) is queried.
212
Colette Rolland et al.
the Holbrook’s chunk based on use scenarios with the Define analysis model chunk supporting construction of an analysis model out of scenarios. Even short and schematic, this example illustrates the use of SgmlQL queries for retrieving scenario base chunks meeting the requirements of a user. It suggests at least two comments: (1) the need for a thesaurus to support an efficient query process and (2) the possibility to capitalise from experience, for instance, by inserting in the method base the new approach when fully generated. This insertion can be done using specific commands provided by SgmlQl.
6. Conclusion In this paper we propose an approach for supporting the reuse of scenario based chunks made available in a scenario method base. The motivation for developing such an approach was twofold: first, there exists a large corpus of available scenariobased approaches that has not been formalised yet, and secondly there is an explicit need for incorporating scenario-based approaches in existing methods. FUSION, OMT and UML are examples of such enhancements that the method engineers would like to reproduce. The proposed approach advocates a modular representation of scenario chunks and an intentional description of their reuse context. The former results is cohesive chunks which are applicable in specific situations for specific purposes whereas the latter provides contextual information identifying in which specific design situations for which specific design intentions the chunks are reusable. The paper also reports on the implementation of a scenario method base in SGML and illustrates the reuse process through an example. Future work shall concentrate on developing guidelines to integrate scenario chunks in existing methods and the implementation of these guidelines in the Sgml context. Besides, in order to support the process for retrieving chunks matching specific requirements we are developing a set of SgmlQL macro-queries. Finally we shall work on an HTML presentation of the query responses.
7. Références 1. Booch, J. Rumbaugh, I. Jacobson, «Unified Modeling Language », version 1.0, Rational Software Corporation, Jan. 1997. 2. Bubenko, C. Rolland, P. Loucopoulos, V. De Antonellis, « Facilitating ‘Fuzzy to Formal’ Requirements Modelling », Proc. of the First International Conference on Requirements Engineering, April 1994, Colorado Springs, Colorado. 3. Carroll, « The Scenario Perspective on System Development », in J.M. Carroll (ed.), Scenario- Based Design: Envisioning Work and Technology in System Development (1995).
Specifying the Reuse Context of Scenario Method Chunks
213
4. Cockburn, «Structuring use cases with goals», Technical report, Human and Technology, 7691 Dell Rd, Salt Lake City, UT 84121, HaT.TR.95.1, http://members.aol.com/acocburn/papers/ usecases.htm (1995). 5. Coleman, P. Arnold, S. Bodoff, C. Dollin, H. Gilchrist, F. Hayes, P. Jeremaes, «Object-Oriented Development : The FUSION Method », Prentice Hall, 1994. 6. V. De-Antonellis., B. Pernici, P. Samarati, (1991) «F-ORM METHOD: A Methodology for Reusing Specification», In Object Oriented Approach in Iforamtion Systems, Van Assche F., Moulin b., Rolland C. (eds), North Holland, 1991 7. T. Erickson, «Notes on Design Practices: Stories and Prototypes as Catalysts for Communication», in J.M. Carroll (ed.), Scenario- Based Design: Envisioning Work and Technology in System Development (1995). 8. D. Filippidou, P. Loucopoulos, «Using Scenarios to Validate Requirements in a Plausibility-Centred Approach», Proc. Of the 9Th Conference on Advanced Information Systems Engineering, Barcelona, Catalonia, Spain, June 1997. 9. M. Glinz, «An Integrated Formal Model of Scenario based on Statecharts», Lecture Notes in Computer Science’95,pages 254-271, 1995. 10. C. F. Goldfarb, « The SGML Handbook », Oxford Clarendon Press, 1990. 11. C. H. Holbrook_III, « A Scenario-Based Methodology for Conducting Requirement Elicitation», ACM SIGSOFT Software Engineering Notes, 15(1), pp.95-104, 1990. 12. Hsia P, Samuel J, Gao J, D., Toyoshima, Y. and Chen ,C . (1994) «Formal Approach to Scenario Analysis», IEEE Software, 11, 33-41 13. I. Jacobson, M. Christerson, P. Jonsson and G. Oevergaard, «Object Oriented Software Engineering: a Use Case Driven Approach», (Addison-Wesley, 1992). 14. I. Jacobson, «The Use Case Construct in Object-Oriented Software Engineering», in John M. Carroll (ed.), Scenario-Based Design: Envisioning Work and Technology in System Development (John Wiley and Sons, 1995) 309-336. 15. I. Jacobson, M. Ericsson and A. Jacobson, «The Object Advantage, Business Process Reengineering with Object Technology» (Addison-Wesley Publishing Company, 1995). 16. M. Jarke, K. Pohl, P. Haumer, K. Weidenhaupt, E. Dubois, P. Heymans, C. Rolland, C. Ben Achour, C. Cauvet, J. Ralyte, A. Sutcliffe, N. A. Maiden and S. Minocha, «Scenario use in European software organisations - Results from site visits and Questionnaires», Esprit Reactive Long Term Research Project, 21.903 CREWS, Deliverable W1 : Industrial Problem Capture Working Group, 1997. 17. P. Johnson, H. Johnson and S. Wilson, «Rapid Prototyping of User Interfaces driven by Task Models», in John M. Carroll (ed.), Scenario-Based Design:
214
Colette Rolland et al.
Envisioning Work and Technology in System Development (John Wiley and Sons, 1995) 209-246. 18. J. Karat, «Scenario Use in the Design of a Speech Recognition System», in J.M. Carroll (ed.), Scenario- Based Design: Envisioning Work and Technology in System Development (1995). 19. M. Kyng, Creating Contexts for Design, in John M. Carroll (ed.), «ScenarioBased Design: Envisioning Work and Technology in System Development» (John Wiley and Sons, 1995) 85-107. 20. V. Lalioti and B.Theodoulidis, «Use of Scenarios for Validation of Conceptual Specification», Proceedings of the Sixth Workshop on the Next Generation of CASE Tools, Jyvaskyla, Finland, June 1995. 21. J.C.S. do Prado Leite, G. Rossi, F. Balaguer, A. Maiorana, G. Kaplan, G. Hadad and A. Oliveros, «Enhancing a Requirements Baseline with Scenarios», In Third IEEE International Symposium On Requirements Engineering (RE'97), Antapolis, Maryland (IEEE Computer Society Press, 1997) 44-53. 22. J. Lemaitre, E. Murisasco, M. Rolbert, SgmlQL, «Un langage d'interrogation de documents SGML», Proceedings of the 11th conference on Advanced DataBases, August 1995, Nancy, France. 23. J. Nielsen, Scenarios in Discount Usability Engineering, in John M. Carroll (ed.), «Scenario-Based Design: Envisioning Work and Technology in System Development» (John Wiley and Sons, 1995) 59-85. 24. V. Plihon, C. Rolland, «Modelling Ways-of-Working» Proc. Of the 7th Int. Conf. On «Advanced Information Systems Engineering», (CAISE), Springer Verlag (Pub.), 1995. 25. C. Potts, K. Takahashi and A.I. Anton, «Inquiry-based Requirements Analysis», in IEEE Software 11(2) (1994) 21-32. 26. N. Prat, «Goal Formalisation and Classification for Requirements Engineering», Proceedings of the Third International Workshop on Requirements Engineering: Foundations of Software Quality REFSQ’9 , Barcelona, june 1997. 27. R. Prieto-Diaz, P. Freeman, « Classifying Software for Reusability», IEEE Software, Vol 4 No 1, 1987. 28. B. Regnell, K. Kimbler and A. Wesslen, «Improving the Use Case Driven Approach to Requirements Engineering», in the Second IEEE International Symposium On Requirements Engineering, York, England (I.C.S. Press, March 1995) 40-47. 29. S.P. Robertson, «Generating Object-Oriented Design Representations via Scenarios Queries», in John M. Carroll (ed.), Scenario-Based Design: Envisioning Work and Technology in System Development (John Wiley and Sons, 1995) 279-308.
Specifying the Reuse Context of Scenario Method Chunks
215
30. C. Rolland, G. Grosz, «A General Framework for Describing the Requirements Engineering Process», IEEE Conference on Systems Man and Cybernetics, CSMC94, San Antonio, Texas, 1994. 31. C. Rolland, N. Prakash, «A proposal for Context-Specific Method Engineering», IFIP TC8 Working Conference on Method Engineering, Atlanta, Gerorgie, USA, 1996 32. C. Rolland, C. Ben Achour, C. Cauvet, J. Ralyte, A. Sutcliffe, N.A.M. Maiden, M. Jarke, P. Haumer, K. Pohl, E. Dubois and P. Heymans, «A Proposal for a Scenario Classification Framework», ESPRIT Reactive Long Term Research Project 21.903 CREWS, Deliverable I1: Initial Integration Workpackage (1997). 33. J. Rumbaugh, M. Blaha, W. Premerlani, F. Eddy, and W. Lorensen, «ObjectOriented Modeling and Design», (Prentice Hall, 1991). 34. J. Rumbaugh, «Getting started, using use cases to capture requirements», Journal of Object Oriented Programming, Sept. 1994. 35. J. Rumbaugh and G. Booch, «Unified Method», Notation Summary Version 0.8 (Rational Software Corporation, 1996). 36. K.S. Rubin and A. Goldberg, «Object Behaviour Analysis», Communications of the ACM 35(9) (1992) 48-62. 37. B. Thomé, «Systems Engineering : Principles and Practice of Computer-based Systems Engineering», in B. Thomé (ed), John Wiley & Sons (1993). 38. R. Wirfs-Brock, «Designing Objects and their Interactions: A Brief Look at Responsability-driven Design», in John M. Carroll (ed.), «Scenario-Based Design: Envisioning Work and Technology in System Development» (John Wiley and Sons, 1995) 337-360. 39. M. R. Young, P. B. Barnard, «The Use of Scenario in Human-Computer Interaction Research: Turbocharging the tortoise of Cumulative Science», CHI + GI 87 Human Factors in Computing Systems and Graphics Interface, Toronto, 1987.
8. Appendices 8.1 Appendix 1 : Characterising Scenarios In [32] a framework for scenario classification is proposed to characterise scenarios according to a certain number of facets which are grouped into views. Four views have been identified : 1. the form view, 2. the contents view, 3. the purpose view and 4. the life cycle view.
216
Colette Rolland et al.
The form view answers the question ‘in which form is a scenario expressed ?’. The response is provided through two facets namely the description facet and the presentation facet. • The description facet characterises the level of formality and the medium used for the scenario description. Texts, graphics, images, videos and software prototyping are examples of media. Note that several media can be used at the same time for describing a scenario. • The presentation facet tells whether a scenario is static or animated, and its interactivity i.e. the capabilities offered to the user to control the way the scenario progresses through time. Consequently, in the descriptor the form view is represented by four properties of scenarios namely, FormDescriptionMedium, FormDescriptionNotation, FormPresentationAnimation and FormPresentation Interactivity. The contents view answers the question ‘what is the knowledge expressed in a scenario ?’. The response is provided through four facets namely the abstraction facet, the context facet, the argumentation facet and the coverage facet. • The abstraction facet indicates whether the scenario is concrete, abstract or mixed. • The context facet explains in which kind of context the scenarios are used. System internal, system interaction, organisational context and organisational environment are examples of contexts where the scenarios can be used. • The argumentation facet indicates whether argumentation concepts are used within the scenarios or not. • The coverage facet indicates the kind of information captured in the scenarios, i.e. whether it is functional, non functional or intentional. The information concerning the structure, the function and the behaviour are qualified as functional, the ones which are tackling performance, time constraints, cost constraint, user support, documentation examples, back up/recovery, maintainability, flexibility, portability, security/safety, design constraints, error handling are non functional and the information concerning goal, problem, responsibility, opportunity cause and goal dependency are said intentional. This leads to the following properties of scenarios in the chunk descriptor : ContentsAbstraction, ContentsContext, ContentsArgumentation and ContentsCoverage. The purpose view answers the question ‘why using a scenario ?’. The response is provided through three criteria. A scenario can be used • in a descriptive purpose, i.e. for describing something which happens in the real world,
Specifying the Reuse Context of Scenario Method Chunks
217
• in a exploratory purpose i.e. for constructing requirements elicitations, or • in a explanatory purpose, i.e. when explanations about the rationale of these issues are required. The purpose perspective is associated in the chunk descriptor to the property called Purpose. The life cycle view is characterised by two facets : the life span facet and the operation facet. It explains ‘how to manipulate scenarios’. • The life span facet indicates whether the scenario is transient or persistent in the RE process. • The operation facet defines if and how scenarios are captured, refined, integrated, expanded, and deleted. This is represented in the chunk descriptor by two properties, namely LifeCycleSpan and LifeCycleOperation. More details on the scenario classification framework and its application on several scenario based approaches can be found in [32]. 8.2 Appendix 2 : The Scenario Method Base Structure
- - -
< ! ATTLIST TARGET TYPE (SCENARIO-BASED | NON SCENARIO_BASED) #IMPLIED> ] >
Change Analysis and Management in a Reuse-Oriented Software Development Setting Wing Lam Department of Computer Science, University of Hertfordshire College Lane, Hatfield, Herts AL10 9AB, UK [email protected], Phone: +44 (0)1707 284337, Fax: +44 (0)1707 284303
Abstract. This paper discusses the need for systematic and methodical approaches for change analysis and management in software projects. We are not concerned with specific techniques (such as program slicing or dependency graphs), but with the overall project-level process of handling change, which we feel is underrepresented in the current literature. The paper describes the efforts being made to manage change at Software Development Services (SDS), a small commercial organisation that specialises in the development of Customer Complaint Systems (CCSs). Because SDS recognises that most change is specific to their domain (CCSs), we are taking a domain-specific approach to change analysis and management. Our main contribution in this paper is a framework for the change process, called the Change Cycle, which is concerned with identifying and formalising reusable change knowledge. The paper reviews current methods and techniques for handling change, and discusses the desirable characteristics of a change process. We then describe the Change Cycle in detail and our experience of using it at SDS. Tool support for change is also outlined, as is our approach to evaluating this work. We conclude that change analysis and management should be treated as an integral part of the software process, not as an adjunct to it.
1. The Need for Systematic Approaches for Handling Change Software systems are not static, but evolve. Fluctuating user requirements, commercial pressures, organisational transition and demands for interoperability (e.g. the Internet) all contribute to volatility in the software process [14, 18]. As [16] noted, “software systems must change and adapt to the environment, or become progressively less useful”. Increasingly then, today’s software engineers need systematic approaches for dealing with change. This paper describes the efforts being made to manage change at Software Development Services (SDS), a small commercial organisation than specialises in the development of Customer Complaint Systems (CCSs). Because CCSs are similar in functionality, SDS has adopted a reuse-oriented software process, where individual CCSs are tailored from a generic CCS prototype. SDS has observed that many of the kinds of changes made to one CCS are similar to the kinds of changes made to other CCSs, i.e. much change is domain-specific. This paper describes how we attempted to exploit this observation in developing a systematic approach to the analysis and management of change at SDS. The format for this paper is as follows. The next section, Section 2, describes the problem of change from the perspective of SDS. Section 3 reviews current methods B. Pernici and C. Thanos (Eds.): CAiSE’98, LNCS 1413, pp. 219-236, 1998. Copyright Springer-Verlag Berlin Heidelberg 1998
220
Wing Lam
for handling change in the literature and indicates how the approach discussed here differs. Section 4 discusses the desirable characteristics of a change process. Section 5 presents our framework for change analysis which we call the ‘change cycle’. Section 6 describes the application of the change cycle to change problems at SDS. Section 7 outlines tool support for change and Section 8 the strategy we are using to evaluate our work. Finally, Section 9 concludes.
2. Change In a Reuse-Oriented Software Process SDS was formed in 1994 to meet the growing market for Customer Complaint Systems (CCSs). The majority of clients for CCSs are retail outlets. In short, a CCS records individual customer complaints in a desired format, tracks the complaint from initiation through to resolution and performs a variety of anlayses over complaint data (e.g. % of complaints resolved within a given time, distribution of complaints relating to a defined service or product group). Like many commercial applications today, CCSs are typically required to run under the Windows environment, either as a standalone or networked (multi-user) application. Because CCSs tend to share similar functionality, SDS has adopted a reuseoriented software process, where an individual CCS for a customer is tailored from a generic CCS prototype. In the SDS software process, change is seen as a central part of the software process of which there are three main categories (Figure 1). 5 3XQ^WU
7U^UbYS 33C
D 3XQ^WU
C`USYVYS 33C
B 3XQ^WU
Figure 1 Types of change 1. 2.
3.
Tailoring changes (T-Changes). This is where SDS identifies with the customer the changes needed to tailor the generic CCS to their specific CCS requirements. Requirement changes (R-Changes). In addition, like many software systems, changes (which are customer-driven) also occurs during the development process (e.g. demonstrations, trials) and after a CCS has been installed on-site. Evolutionary changes (E-Changes). The generic CCS is itself the subject of evolutionary improvements. As SDS develops more CCSs, we expect the generic CCS to gradually become more ‘mature’.
Reuse-Oriented Software Development Setting
221
This paper is concerned primarily with R-Change, which [9] identify as a major challenge to software engineering (use of the word ‘change’ from this point on will refer to R-change). Like many ‘real world’ applications, changing and evolving requirements is a prominent feature of CCS software development projects. As CCSs are highly interactive systems, SDS use a user-centred and prototyping approach to their development. While such an approach is useful in ‘discovering’ customer requirements, we have found that there is little or no control over the change process itself. SDS has observed that the kinds of changes that occur in the development of one CCS are often repeated in the development of other CCSs, i.e. change is domainspecific. As such, one of the goals of SDS is to acquire an understanding of these ‘patterns’ of change. It is envisaged that achieving this will facilitate the development of future CCSs, in particular, helping SDS to develop capabilities in the following areas: 1.
2.
3.
Change anticipation. Anticipate change (e.g. to expect the introduction of new requirements) in order to plan ahead and address change in a pro-active rather than reactive manner. Change categorisation and change strategy reuse. Identify and classify different types of change in order to formulate and re-use strategies for coping with change. Change estimation. Collect cost and time-scale data for types of change in order to aid estimation (e.g. time and cost to implement a change).
These capabilities, however, depend on the capture and reuse of domain-specific change knowledge, which is an aspect of change that is not addressed by the current literature.
3. Current Methods and Techniques for Handling Change There are a number of distinct research areas that provide an angle for addressing aspects of software evolution, some of which are particularly relevant to change at the requirements level. •
Process models for software evolution attempt to define high-level process models that attempt to either reduce or eradicate the perceived gap between software development and maintenance. The IEEE software maintenance model, described by [3], proposes a seven-step ‘waterfall-like’ process: a) problem identification, b) analysis, c) design, d) implementation, e) system test, f) acceptance test and g) delivery. This, however, provides no real detailed guidance for the software maintainer. The FEAST (Feedback and Software Technology) model [16] is an advancement of Lehman’s well-known E-process model. In the FEAST model, the software process is modelled in a dynamic fashion as a continuous system varying over time. Execution of the process is through control loops, which produce ‘feedback’ to drive the next steps in the process. However, FEAST is a theoretical (rather than practical) model, and questions about the
222
•
•
•
•
•
Wing Lam
level of modelling (fine-grain modelling is likely to result in extremely complicated models) and nature of control loops (open or closed) are unclear. Heuristic support includes guidelines for managing change, which might be appropriate at different levels of support, e.g. managerial-level, project-level, and task-level. [22] identify 3 strategies for change: a) identifying change early in the lifecycle, b) facilitating the incorporation of change and c) reducing change. They offer further guidelines about techniques appropriate to each strategy, e.g. the use of prototyping, simulation, work-throughs and scenarios in strategy a). Evolutionary delivery methods [7] encourage incremental development and early feedback. Evolutionary delivery methods do, however, rely on being able to compartmentalise the system and to deliver it in stages. The application of evolutionary delivery methods has generally been used at a macro-level to deal with change from a user-perspective, such as in the form of prototyping. However, support for other aspects of change such as studying the impact of change or change anticipation is not addressed. Logic languages can help to reason formally about change and impact [24, 4]. [4] describes a language with goal-structures that capture the relationships between goals and sub-goals, which can be used to reason about alternative design decisions and analyse trade-offs. One problem here is that there are likely to be many influences on the software process, and a language that attempts to capture all these concepts is likely to be both large and complex (even before one attempts to apply it). In addition, there are validation issues as the ability to do any useful reasoning relies on having realistic and representative models of the reallife process. Taceability attempts to define the important associative relations between and across actors and artefacts in the software process [20. Traceability is important in determining how change to one software artefact might affect another artefact. Traceability at the implementation level is supported by techniques such as program slicing [24] and program dependence graphs [19]. However, there is an increasing need to extend traceability to earlier levels of software engineering. One issue is the granularity at which traceability is performed. For example, in requirements engineering, traceability might be performed at the requirements document level and/or at the level of individual requirements. Environments have been proposed to support change impact and change propagation. Environments tend to support a mixture of automatic and user-assisted change operations. Such environments operate at the implementation level and use program dependency graphs [8, 11]. An expert system environment has also been suggested by [1].
The work so far in this area concentrates on general and non-domain-specific methods and techniques for supporting change. Our work differs, in that we are concerned with domain-specific methods for supporting change in the software process. It is generally recognised that domain knowledge is central to the enactment of many software processes, e.g. requirements engineering [6]. Our general hypothesis here is that individual application domains exhibit “patterns” of change (particularly at the requirements level), and that understanding these change patterns can facilitate the evolution of future systems in the domain, as well as provide guidance for the application of non-domain-specific techniques. In short, we are assuming that change in
Reuse-Oriented Software Development Setting
223
the past (historical change) will be indicative of change in the future. This is not an unusual assumption in software engineering, e.g. historical data forms the basis for reliable estimation models [21]. In the following, we present a view of the change process, called the change cycle, which we have developed in our current work. First, however, we feel it is beneficial to discuss what the general, desirable characteristics of a change process are.
4. Desirable Characteristics of a Change Process A prescriptive process describes the activities necessary to complete a goal and the order in which those activities are to be carried out [25]. [17] describes some of the desirable properties of a prescriptive process to be: convey information readily, have reasons supporting their definition, have formally defined syntax and semantics, comprehensive, describe and integrate different levels of abstraction and is reusable. One approach for coping with change is for organisations to develop prescriptive models of the change process. From our discussion in the previous sections, a prescriptive model of the change process should help in some, if not all, of the following ways: • • • •
Help in change prevention. Not all change can or should be prevented. However, it is sensible to prevent or least minimise unnecessary or ‘risky’ change. Help in impact analysis. It should be possible to reason, or at least conjecture, about the potential effects of change on software artefacts and the software process. Help in change planning. Change should be planned into the software process rather than being treated as an adjunct to it. Also, the sequencing of change and the collective effect of change over change in isolation should be considered. Help in stability assurance. After a change has been implemented, assurance procedures should exist to ensure that integrity (software and documentation) is maintained. This is particularly important in the case of safety-critical systems.
Ultimately, dealing properly with change at the time will save on re-work further down the software time-line, just as dealing with problems at the requirements level is more cost-effective than dealing with the problem at implementation.
5. The Change Cycle: A Framework for Change Analysis A framework for change analysis that we have developed is the change cycle (Figure 1). The change cycle provides a concise and integrated view of the change process that emphasises the domain-specific and reuse perspectives that are important in our work.
224
Wing Lam SXQ^WU
DbQSUQRY\Ydi 1^Q\icYc
cSU^QbY_c
T_Se]U^d
SXQ^WU
QbSXYdUSdebU
QddbYRedUc _bWQ^YcQdY_^ c`USYVYS
3XQ^WU 3Q`debU
T_]QY^ c`USYVYS
T_Se]U^d TU`U^TU^SYUc
`b_SUcc
UhU]`\Qbc
Y]`QSd cdeTi
SXQ^WU [^_g\UTWU
Y]`QSd
VUUTRQS[
bUecU
SXQ^WU
QccUcc]U^d
ecU SQcUc `b_ZUSd c`USYVYS
3XQ^WU =Q^QWU]U^d
T_]QY^ c`USYVYS
SXQ^WU
SXQ^WU
ecU SQcU
QddbYRedU
bUecU
WU^UbQ\YcQdY_^
BUecU 1^Q\icYc
Figure 2 The change cycle At the centre of the change cycle is an evolving body of change knowledge. The change cycle is composed of four distinct sectors which contribute to the change knowledge: 1. 2. 3. 4.
Traceability analysis. An analysis of the dependencies between and across software artefacts and actors in the software process. Change capture. The modelling and capture of change on specific projects. Reuse analysis. The derivation of generic change patterns and reusable change knowledge. Change management. The application of generic change patterns and reusable change knowledge in the management of change on a specific project.
Steps 2-4 share a similar overall process to that of domain-specific reuse approaches, for example, the reuse of domain-specific requirements as described by [12]. We argue that this should not be surprising, as requirements engineering, like change analysis and management, is a process that is strongly guided by domain expertise [6, 5]. The change process in Figure 2 is a cycle because each iteration of the change cycle refines and re-uses the body of change knowledge. We view change knowledge as something which evolves over time and with development experience in the domain. It is both unreasonable and unrealistic to expect a single analysis of change to miricuously produce a complete and definitive knowledge about change, especially in complex real-world domains (as opposed to academically-bounded domains often used in the literature). Each iteration of the change cycle, however, can contribute to an explicitly documented body of change knowledge. One role of the process cycle is to organise specific change analysis and management techniques within the broader change process. Table 1 outlines some of the specific techniques which can be used in each sector of the change cycle. The
Reuse-Oriented Software Development Setting
225
‘Focus of techniques’ field in Table 1 describes the general purpose of the set of techniques; the ‘Level/scope of usage’ indicates the intended scope of these techniques. Our list of techniques is not complete, and refer to the ones which we have used so far in our work. [22] also describe a more general set of techniques which could also be fitted into the context of the change cycle). Table 1 Techniques in the change cycle Focus of techniques
Techniques
Traceability analysis
Model associative relationships in the existing software process.
Change capture Reuse analysis
Acquisition of change knowledge through the analysis of exemplars. Identification and formalisation of reusable change knowledge.
Change management
Application of reusable change knowledge and the provision of feedback for its improvement.
The grounding for many of these techniques should be familiar to those with an appreciation of basic software engineering methods, e.g. use-case modelling, metrics and traceability. We believe that a systematic process for managing change can be achieved by combining and applying these basic software engineering concepts. We provide an illustration of the use of some of these techniques at SDS in the next section.
6. Application at SDS Because the CCS domain is real-world domain that is subject to real change and real customers, we feel the domain is a suitable domain upon which to concentrate our research efforts on. In addition, the domain is relatively intuitive (compared to, for example, the author’s previous work on aircraft control systems [12]), so the learning curve for understanding the domain did not adversely impede our research. We have currently performed one iteration (which took about one person-week of effort) of the change cycle to study change in the customer complaints domain at SDS. The result of this iteration is an initial body of change knowledge, which we will refine in future iterations. We intend to perform one iteration per CCS developed at SDS, in order to refine our change knowledge at every opportunity. In the following, we describe the application of specific techniques in each sector of the change cycle. However, instead of burdening the reader with the fine details of our application, we draw out the main goal-task activities in each sector.
226
Wing Lam
6.1 Sector 1: Traceability analysis Goal: Study documentation traceability Task: Draw the documentation architecture Just as an implementation has an architecture, so do requirements and designs. The importance of the requirements architecture in requirements reuse is described by [15]. The requirements and design architecture is often reflected in the documentation architecture. The documentation architecture describes the internal structure of system documentation and the dependency relationships that exist between documentation units within the same document and across different documents. At SDS, a CCS requirements document is produced from the generic CCS. From this, a CCS design document is produced. Figure 3 shows the documentation architecture. Here, the small boxes depict the documentation units, and the arrows the dependency relationships between units that we have uniquely identified. XYWX \UfU\ Ve^SdY_^Q\ bUaeYbU]U^dc
T!
T"
ecUb Y^dUbVQSU
42 dQR\U
T#
cSbUU^
T!!
bUaeYbU]U^dc
bUaeYbU]U^dc
T$
cdbeSdebUc
]UTYQ
T'
TQdQ
cSbUU^
bUaeYbU]U^dc
T(
T%
fYceQ\ bUaeYbU]U^dc
33 C B BUa Uaee YbU YbU] ] U^d U^dcc 4_Se 4_Se] ] U^d
T&
bU`_bd
\Qi_edc
T)
]_Te\U c`USYVYSQdY_^
c`USYVYSQdY_^
T! bU`_bd UhU ]`\Qbc
3 3 C 4UcY 4UcYWW ^ 4 _Se _Se] ] U^ d
Figure 3 The documentation architecture Goal: Examine the nature of documentation dependencies Task: Instantiate documentation dependency templates We used documentation dependency templates (DDTs) to help examine the nature of dependencies in the documentation architecture, i.e. make clear why there is a dependency not just the fact that there is. An example of an instantiated DDT is shown in Table 2.
Reuse-Oriented Software Development Setting
227
Table 2 An instantiated DDT Documentation dependency Documentation structural unit 1 Documentation structural unit 2 Dependency Consistency rules
D7 Data-requirements-CCS-requirements-document DB-table-structures-CCS-design-document Requirements to DB-table-structure. 1. Data requirements must match the underlying database table structure. 2. Data requirements must be able to be met through processing from data stored according to the underlying database table structure.
We instantiated DDTs for each dependency identified in the document architecture to understand, at a fine-grain level, the traceability concerns in CCS development at SDS.
6.2 Sector 2: Change capture Goal: Elicit the change process followed by developers for dealing with specific kinds of change Task: Walk-through change scenarios An important element of change capture was to understand how staff as SDS deal with different kinds of change. We elicited processes for dealing with change by identifying and walking-through different change scenarios. We identified common change scenarios based on an examination of the kinds of changes that had occurred in previous CCSs. We used event traces to represent and discuss change scenarios because of their ease of understanding and semi-rigorous approach. Figure 4 shows an example of a change scenario in the case of a change to a screen layout (i.e. visual requirement). C4C 3ecd_]Ub 9^dUbVQSU
Y]`\YSQdY_^c _RdQY^ Secd_]Ub QWbUU]U^d bU TUcYW^ cSbUU^ `b_d_di`U TU]_^cdbQdY_^
Figure 4 A change scenario
e`TQdU bUaeYbU]U^dc T_Se]U^d
228
Wing Lam
Goal: Model and capture information about specific changes Task: Characterise the change attributes of a change Change scenarios capture the process of change. To model other features of change, we have identified a set of change attributes, which we formulated during discussion with staff at SDS. We have classified the change attributes into four categories that reflect the aspects of change of most concern to SDS. 1. 2. 3. 4.
Source. Where the change emanates from. Severity. The severity of the change. Effect. The effect that the change has on software artefacts and the software process. Strategy. The strategy (such as process and pitfalls) used to deal with the change on a specific project.
Table 3 shows the change attributes (with examples) we have identified so far, but which we acknowledge may not be complete. Table 3 Change attributes Attribute Category What Source
Attribute
Example Attribute Values Add ‘counter’ display to the new complaints form. John Smith
Effect
Description Who (made the change request) When Why Time Cost (internal) Change artefacts
Strategy
Process
Severity
Pitfalls
Prototype demonstration, 12-10-96 Counter needed to display number of unresolved complaints 5 person-hours £125 at £25/person-hour 1. Screen (visual) requirements (CCS Requirements document) 2. Data requirements (CCS Requirements document) 3. Screen layouts (CCS Design document) 4. New module specification (CCS Design document) 5. New complaints form (code) 6. New function in function library (code) 1. Re-design new screen layout. 2. Design counter function. 3. Implement new screen. 4. Implement new function. 5. Test 1. 2. 3. 4.
Adding a new field may require a significant redesign to the original form. Adding a new field may violate screen design consistencies. New field of this type (counter) should ‘match’ existing fields of the same type. Counter fields should not add or modify data in the underlying database.
Reuse-Oriented Software Development Setting
229
Finding the values for change attributes for a particular change takes place over the life of a change, i.e. from the inception of the change through to its completion. For example, the severity of a change in terms of its time and cost can only be known once the change has been completed (though we might have some idea beforehand of its likely severity — this is an example of the kind of reusable knowledge we are trying to exploit). It should be recognised that what we are doing here is taking a direct modelling approach, i.e. we view change as an explicit concept in the software process just as we view each requirement or each module of code as an explicit concept. Each change in the development of a CCS can be captured in terms of our change attributes. This provides SDS with a convenient way of documenting change, but also enables SDS to build up an empirical and historical record of change upon which to generalise (sector 4 in our change cycle).
6.3 Sector 3: Reuse analysis Goal: Establish a set of reusable change patterns Task: Build-up a change use-case hierarchy Through discussions at SDS over many concrete change scenarios, we were able to identify common change scenarios which we call change use-cases (as in the usecases described by [10]). Change use-cases are reusable across CCS projects. Change scenarios are instances of a change use-case. For example, our discussions at SDS quickly revealed three common change use-cases: 1.
2. 3.
Visual-change-use-case. Changes to the screen layout (visual requirements), which are often raised during prototype demonstrations with the customer during an iterative development process. Report-change-use-case. Change to reports, especially when ‘example’ reports are produced from test data and are given to the customer for inspection. Data-change-use-case. Change to data requirements, which often includes the inclusion of new information to be recorded by the system (customers tend not be know at the start what their exact data requirements are).
We suspect that there are many more change use-cases that we have not yet identified. In particular, there are also specialised forms of the change use cases mentioned already. A report change use-case, for example, may have a report-layout-change-usecase and a new-data-change-use-case. Building up a complete picture of these change use-cases in the customer complaints domain in terms of a change use-case hierarchy (Figure 5), is part of our on-going work.
Figure 5 Evolving a change use-case hierarchy We develop a generic event trace for each change use-case by abstracting, using manual inspection methods, from concrete event traces. The change use-case thus captures reusable process knowledge for dealing with a particular type of change. Goal: Create reusable change knowledge Task: Abstract from change attributes The analysis of many similar change exemplars (similar in terms of being instances of the same change use-case) allows us to define general or typical values for change attributes. The change attributes we have initially focussed on are those in the severity category i.e. time and cost of change (e.g. a data-change will typically take between 15-20 man-hours to complete). It should be noted that productivity models are based on historical data [21]. We are essentially doing a similar thing, but at an earlier stage in the metrics collection process. To give an example, customers often ask for changes to a CCS after it has been installed. One problem for managers at SDS is estimating what the time and cost of the change will be. Presently, SDS relies on management expertise to carry out estimation. However, we can now facilitate this estimation process by providing typical time and cost figures based on past change exemplars. We expect the reliability of these figures to improve over time as the sample size for the number of changes increases. Goal: Study the impact of change to software documentation Task: Extend traceability in change scenarios The change scenarios described earlier help to elicit the change process carried out by staff at SDS. We have found that change scenarios are also a good way of validating change processes against the documentation architecture. For example, when a developer at SDS says ‘update requirements document’ in a change scenario, we are able to question what part of the requirements document, i.e. the documentation unit, and then use documentation dependencies to ascertain what checks need to be done on related parts of the documentation. If we had not done this, changes addressed and documented ‘locally’ but not in other dependent parts of the document architecture may leave the documentation in an inconsistent state. From this we are able to formulate a number of change heuristics of the form:
Reuse-Oriented Software Development Setting
231
WHEN change IS A screen-change THEN UPDATE screen-requirements IN CCS-requirements-document AND REQUIREMENTS-CHECK user-interface-requirements IN CCSrequirements-document AND REQUIREMENTS-CHECK high-level-functional-requirements IN CCSrequirements-document AND DESIGN-CHECK screen-layouts IN CCS-design-document… Change heuristics formalise the ‘what’ to change, ‘what’ to check and ‘when’ to check. We see change heuristics as essential in system maintenance situations where the original CCS developer is not the same person who carries out the maintenance. We also believe that change heuristics (though not necessarily of the same form as here) are central to providing knowledge-based support for the change process.
6.4 Sector 4: Change management Goal: Manage change on a specific project Task: Tailor the generic change management process Change management is the fourth sector of the change cycle and is concerned with the management of change on a specific project. The general management process is outlined in Figure 6, which begins with a change request from the customer. 3ECD?=5B
SXQ^WU bUaeUcd dUcd Q^T bU\UQcU
TUdUb]Y^U di`U _V SXQ^WU
e`TQdU T_Se]U^dQdY_^
SXUS[ SXQ^WU Y]`\YSQdY_^c
SXQ^WU
TU`U^TU^SYUc
ecU SQcUc
QTTbUcc SXQ^WU Y]`\YSQdY_^c
T_Se]U^dQdY_^
di`YSQ\
TU`U^TU^SYUc
SXQ^WU QddbYRedUc
UcdY]QdU S_cd Q^T UVV_bd
TU`U^TU^SYUc SXQ^WU ecU SQcUc
381>75 ;>?G<5475
S_^VYb] gYdX Secd_]Ub
UhUSedU SXQ^WU
^UW_dYQdU gYdX Secd_]Ub `\Q^ SXQ^WU
Figure 6 General change management process At the centre of the change management process is the body of change knowledge which is at the centre of the change cycle. We propose that projects tailor the change management process for specific projects in order to define a standard change man-
232
Wing Lam
agement process. Such processes can be used to provide practical guidance for dealing with change, especially when encoded within a process-driven softwareengineering environment. Goal: Resolve viewpoint problems of change Task: Issue and use a ‘Change processes’ document One of the problems that we have come across is that change is viewed differently from different viewpoints. For example, the ‘salesperson’ that interacts with the customer might consider a change to be relatively ‘easy’. However, the same change might be considered extremely difficult from a developer viewpoint, e.g. because the developer has made some (not unreasonable) assumption. In this kind of case, there is a mismatch between the individual perceptions of change. Resolving viewpoint problems requires working towards a common understanding of change processes between members of a project. One approach is to produce and use a ‘change processes’ document detailing the kind of change knowledge that we have described in this paper (change use-cases, change attributes etc.). Such a document could be used as a reference document in different scenarios, e.g. change estimation and customer negotiation.
7. Tool Support for Change Captured change knowledge can be exploited in automated or semi-automated tool support for change. We are currently investigating the architectural requirements for tool support for change in SDS. A proposed ‘high-level’ architecture for a change environment, based on our work with the change cycle at SDS, is shown in Figure 7. There are two types of users, one type who is concerned with the input of change knowledge (post project perhaps), and the other type who is looking for help during a current project. SX Q^WU
SX Q^WU
SX Q^WU
[^_g\UTWU
[^_g\UTWU
WeYTQ^ SU
Y^`ed d__ \c
RQcU
d__\c
T_Se]U^d QbSXYdUSdebU `b_SUcc
^Udg_b[
WeYTUb
UTYd_b
`b_SUcc cdU`c
T_Se]U^d TU`U^TU^SYUc UfU^d dbQSUc
TYQ\_WeU
[^_g\UTWU Y^`ed
ecU SQcU
SX Q^WU
]Q^ QWUb
UTYd_b
ecU SQcUc
ecUb [^_g\UTWU Y^`ed
SX Q^WU ]QdSXUb
SX Q^WU
]Q^ QWUb
di`Uc WU^UbQ\
SX Q^WU
SX Q^WU
V_b]c
QddbYR edUc
UcdY]Qd_b
_b di`YSQ\ TQdQ
Figure 7 Proposed high-level architecture for a change environment
ecUb `b_ZUSd ]U]RUb
Reuse-Oriented Software Development Setting
233
The change-environment is comprised of three kinds of tools. First, change knowledge input tools which allow the input of change data and knowledge, such as a network editor for ‘drawing’ out a document architecture. Second, a change knowledge base (KB) for storing change knowledge (though we have not yet worked out the exact representation for the KB). Third, change guidance tools that are able to process change knowledge and provide guidance on a specific project. We see three main change-guidance tools: a change matcher, process guider and estimator. The change matcher matches helps the user match the current change to a change use-case in the change KB. The process guider assists the user by proposing steps for dealing with the change. These process steps are derived from the event trace in the change usecase and by tracing documentation dependencies in the document architecture. The estimator is a tool that is able to generalise from a large collection of change instances. For example, one function of the estimator would be to produce the average time it takes to complete a particular kind of change.
8. Evaluation Approach Our evaluation approach is loosely based around a Goal-Question-Metric [2] paradigm, and on two kinds of cycle which ‘track’ the process cycle (Figure 8).
9]`b_fU]U^d 3iS\U A! =!
FQ\YTQdY_^ 3iS\U
@b_SUcc 3iS\U
A$
=%
=" =$ =# A"
A#
Figure 8 Improvement and validation cycles The two kinds of cycle are: 1.
2.
Improvement cycle. This is concerned with the establishment of improvement goals. Typical goals that we have found relevant include (G1) reduction in the overall level of change, (G2) reduction in the level of change for a particular type of change, (G3) faster turnaround in completing change requests from the customer. Validation cycle. This is concerned with the establishment of metrics or indicators (because it might be difficult to metricate some aspect of an improvement goal) that supports the improvement goals. Metrics and indicators include (M1) total number of changes (supports G1), (M2) total number of changes pertaining
234
Wing Lam
to a particular change use-case category (supports G2) and (M3) average effort taken to complete a change (supports G3). At the time of writing, we have just initiated the collection of metric data. However, we have found that establishing appropriate metrics for change is in itself a nontrivial problem for which there is little discussion of in the current literature. The identification of appropriate and meaningful (to SDS) metrics is part of our on-going work. Our initial experiences suggest that insightful metrics on change is unlikely to be supported without established change procedures in place within the organisation, as exhibited by detailed change request forms and change monitoring tools (as in the change environment proposed earlier). Essentially, we can not do any detailed reasoning on change if the change data is not there in an amenable form. Part of our strategy here has been to choose change attributes that reflect the kind of analysis that we wish to perform.
9. Conclusions This paper has described efforts at change analysis and management in a commercial setting. The starting point for our work was the fact that the systems being developed all pertained to the same domain (namely, the CCS domain). This has encouraged us to take a domain-specific approach to change analysis and management, which differs in emphasis from the existing work on change we reviewed in Section 3. Our main contribution to this area is the ‘change cycle’, which attempts to provide a concise and integrated view of the change process. We have shown how the change cycle has been applied to address aspects of the change problems in the CCS domain at SDS. One area that we feel we need to elaborate more on in our work is the relationship between the change process and perceived models of the software process, such as Waterfall or Prototyping. This is part of our on-going work to develop a ‘changeenhanced’ version of a prototying-centred software development approach. As we noted in Sections 7 and 8, tool support for the change process and change metrics are two further areas of on-going work which we hope to report on in more detail in future papers.
10. References 1.
2.
3.
Avellis, G. (1992), CASE support for software evolution: A dependency approach to control the change process. In proceedings of 5th International Workshop on Computer-aided software engineering, pp.62-73, Montreal, Canada, July 1992. Basili, V. and Weiss, D. (1984), A method for collecting valid software engineering data, IEEE Transactions on Software Engineering, November 1984, pp.728-735. Bennett, K. (1996), Software evolution: past, present and future, Information and software technology, 38:673-680, 1996.
Reuse-Oriented Software Development Setting
4. 5. 6. 7. 8.
9.
10. 11. 12. 13.
14.
15. 16. 17. 18.
19.
20. 21. 22.
235
Chung, L., Nixon, B. and Yu, E. (1997), Dealing with change: an approach using non-functional requirements, Journal of Requirements Engineering, 1(4), 1997. Curtis, B., Kellner, M.I. and Over, J. (1992), Process modelling, Communications of the ACM, 35(9), 1992. Fickas, S. and Nagarajan, P. (1988), Critiquing software specifications, IEEE Software, November, 1988. Gilb, T. (1988), Principles of Software Engineering Management, AddisonWesley, England. Han, J. (1997), Supporting impact analysis and change propagation in software engineering environments, In Proceedings of the 8th IEEE International Workshop on Software Technology and Engineering Practice, London, UK, 14-18 July, 1997. Harker, S., Eason, K. and Dobson, J. (1993), The change and evolution of requirements as a challenge to the practice of software engineering, In proceedings of the IEEE international symposium on requirements engineering (RE’93), San Diego, California, 1993. Jacobson, I., Griss, M and Jonsson, P. (1997), Software reuse: architecture, process and organisation for business success, ACM Press, New York. Kaiser, G., Feiler, P. and Popovic, S. (1988), Intelligent assistance for software development and maintenance, IEEE Software, 5(3):40-49, May 1988. Lam, W. (1997), Achieving Requirements Reuse: a Domain-Specific Approach from Avionics, Journal of Systems and Software. 38(3): 197-209, 1997 Lam, W., McDermid, J.A. and Vickers, A.J. (1997), Ten Steps Towards Systematic Requirements Reuse (expanded version), Journal of Requirements Engineering, 2:102-113, 1997. Lam (1998a), Managing requirements evolution and change, IFIP WG2.4 Working conference: Systems implementation 2000: languages, methods and tools, Berlin, Germany, February 23-26, 1998. Lam, W. (1998b), A Case-study of Requirements Reuse through Product Families, Annals of Software Engineering (to appear). Lehman, M. (1996), Feedback in the software evolution process, Information and Software technology, 38:681-686, 1996. Madhavji, N. (1991), The Process Cycle, Software Engineering Journal, September, 1991. Madhavji, N. (1997), Panel session: impact of environmental evolution on requirements changes, In Proceedings of the 3rd IEEE International Conference on Requirements Engineering, 1997. Podgurski, A. and Clarke, L. (1990), A formal model of program dependencies and its implication for software testing, debugging and maintenance, IEEE Transactions on Software Engineering, SE-10(4):352-357, 1984. Pohl, K., Domges, R. and Jarke, M. (1997), Towards method-driven trace capture, 9th International Conference, CaiSE’97, Barcelona, Spain, June 1997. Putman, L. and Myers, W. (1992), Measures for excellence, Reliable software on time, within budget, Yourdon Press, Prentice-Hall, New Jersey, 1992. Sugden, R. and Strens, M. (1996), Strategies, tactics and methods for handling change, In Proceedings of International IEEE Symposium and Workshop on Engineering of Computer-Based Systems (ECBS ‘96), Friedrichshafen, Germany, March 11-15.
236
Wing Lam
23. Yu, E. (1997), Towards modelling and reasoning support for early-phase requirements engineering, In Proceedings of the 3rd IEEE International Conference on Requirements Engineering, 1997. 24. Weiser, M. (1984), Program slicing, IEEE Transactions on Software Engineering, SE-10(4):352-357, 1984. 25. Zave, P. (1986), Let’s put more emphasis on prescriptive methods, ACM Sigsoft Software Engineering Notes, 11(4):98-100.
A Filter–Mechanism for Method–Driven Trace Capture Ralf D¨omges, Klaus Pohl, Klaus Schreck RWTH Aachen, Lehrstuhl Informatik V, 52056 Aachen, Germany email: {doemges|pohl|schreck}@i5.informatik.rwth–aachen.de Abstract: Traceability is a prerequisite for developing high quality (software) systems. Recording and maintaining all available information is too labor intensive and thus by far too expensive. A project-specific definition of the trace information to be recorded and the method fragments (so called trace fragments) to be executed for recording the information provides a solution for this problem. But the amount of traces to be recorded does not only vary from project to project. It also varies between project phases and even within a project phase. As a consequence project-specific trace fragments need to be adapted according to the actual project phase. In this paper we propose a model-based filter mechanism to significantly reduce the required effort to adapt trace fragments. By defining appropriate filters the project manager is able to (dynamically) adapt the project-specific trace fragments to the actual needs. We present an example to highlight the benefits of the approach and discuss possible extensions.
1 Introduction 1.1 Traceability: Definition and Motivation Traceability, more precisely requirements pre– and post–traceability, is recognized by many research contributions and experience reports as an important prerequisite for developing high quality systems (e.g., [Collery, 1988; IEE, 1991; Ramesh et al., 1996]). Among others, traceability facilitates the development, validation and maintenance of a system, provides invaluable support for the integration of changes, reduces the amount of errors during system development, allows experience–based process improvement, and is important to prove the fulfillment of contracts [Jarke et al., 1994; Pohl, 1996b]. Traceability is thus required by various system development standards (e.g., V–Modell [Br¨ohl and Dr¨oschel, 1993], DoD–2167A [DoD-2167A, 1988]) as well as quality and process improvement frameworks, like ISO 9000–3 [ISO, 1991] and CMM [Paulk et al., 1993]. Various contributions (e.g., [Kaindl, 1993; Gotel, 1996; Yu and Mylopoulos, 1994; Conklin and Begeman, 1988]) and traceability frameworks (e.g., [Pohl, 1996b; Ramesh et al., 1996]) define a large set of information to be captured for enabling traceable system development. Obviously, an approach recording all this trace information in each project and for all system components is by far too time consuming, thus too expensive, and therefore likely to fail [Tilbury, 1989]. 1.2 Project–specific Trace Capture: Method–Driven Approach To reduce the effort for capturing and maintaining traceability information the traces, i.e., the information which is actually being stored, has to be recorded according to project–specific needs [Pohl and D¨omges, 1997]. Existing prototypical trace environments (e.g., TOOR [Pinheiro and Goguen, 1996], PRO–ART 1.0 [Pohl, 1996b]) as well as existing commercial requirements and sysB. Pernici and C. Thanos (Eds.): CAiSE’98, LNCS 1413, pp. 237-250, 1998. Springer-Verlag Berlin Heidelberg 1998
238
Ralf Doemges, Klaus Pohl, Klaus Schreck
tem engineering environments like DOORS [Quality Systems & Software, 1996], RDD–100 [Ascent Logic Corporation, 1994], RTM [Marconi Systems Technology, 1996], or SLATE [TD Technologies, Inc., 1996] support the persistent recording and management of trace information. They typically provide a set of generic information types and operations which can be specialized according to project–specific needs. Thus, they empower the project manager to define project–specific trace information, e.g., design decisions and their relations. The environments provide comprehensive consistency checking capabilities and support the retrieval and display of the recorded traces by advanced ad–hoc and pre–defined querying, browsing, and reporting mechanisms. But they do not provide systematic support for recording the defined trace information during the system development process, e.g., the project manager cannot define when (in which situation) a decision should be recorded. Our method–driven approach to trace capture (see [Pohl et al., 1997] for details) overcomes this shortcoming. It supports the explicit definition of project–specific trace information together with trace fragments which define strategies for capturing the information. Moreover the user is guided in recording the information according to the project–specific trace fragment definitions. This is technically achieved by interpreting the defined trace capture strategies in a process–integrated environment. Based on the interpretation the environment reminds the user about the traces to be captured, enforces (e.g., in project critical procedures), and even automates (whenever possible) the recording of traces1. We use the NATURE process meta model [Rolland and Grosz, 1994; Pohl, 1996b] to define the project specific trace fragments. The model distinguishes between: • Atomic fragments for representing the part of a method definition which can be automated. Atomic fragments have no (externally) visible structure and are “hard–coded” in the tool or environment. By executing atomic fragments traces are (automatically) created and recorded in the trace–repository. • Strategy selection fragments for representing the part of the method definition in which the user has to make a decision between at least two alternatives, i.e., alternative trace capture strategies. A trace strategy defines the way how traces are (interactively) created. • Composed method fragments (resp., composed trace fragments) for defining complex trace strategies, i.e., for defining a certain order on a set of trace fragments. The project manager uses the three types of fragments to define the project–specific trace strategies. Thereby s/he (implicitly) defines the trace information to be recorded. To support the definition of the trace strategies we distinguish between four information types. Depending on the information type the recording is performed manually and/or automatically. Table 1 depicts the information types and the way an information is typically being recorded. We have integrated the project–specific trace capture strategy into our TECHMOD [D¨omges et al., 1996] and PRO–ART 2.0 [Pohl, 1996a] environments. We used both environments in small case studies. The studies showed that reminding the stakeholders about the recording of the project specific trace information resulted in traces of higher quality compared to traces produced by following “paper–based” trace capture guidelines. Moreover, recording of unnecessary trace information was avoided and the work load of the users was (significantly) reduced. 1
The detailed mechanism required to enable such a project–specific guidance of trace capture based on the interpretation of method fragments is described in [Pohl and Weidenhaupt, 1997].
A Filter-Mechanism for Method-Driven Trace Capture
information type product information (e.g., ER–model) supplementary product information (e.g., design decisions) process observation information (e.g., method fragment) dependency information (e.g., interrelate ER-model and design decisions)
recording mainly
trace fragment
interactive
process fragment
X
supplementary process fragment
X
process observation fragment dependency fragment
239
automated
X X
X
Tab. 1. Trace information and corresponding trace fragments
1.3 Project–specific Adaptation of Trace Fragments However, it turned out that the trace fragments had to be adapted due to two main reasons: 1. Traces vary between project phases. For example, while in the specification phase recording of structured design decisions was demanded, the maintenance phase focused on recording and interrelating change requests and change approval forms. 2. Traces depend on the components developed, e.g., when safety critical and/or very complex components are developed design decisions, meeting notes, conceptual drawings, as well as scenarios must be captured and interrelated. An obvious solution for the adaptation of trace fragments to project–phase–specific needs was to specialize the (existing) trace fragments accordingly. This solution had two significant shortcomings 1. modeling and/or re–modeling of the required trace fragments was a difficult and time consuming task; 2. programming and/or re–programming of atomic fragments was often required. As a consequence the amount of trace fragments maintained in our method base rapidly increased. The trace fragments were almost identical; they differed only slightly in the trace–capture dependent parts. Thus managing the trace fragments got very complicated and maintaining them consistently was almost impossible. 1.4 Approach and Structure of Paper In this paper we propose a filter mechanism to overcome the above shortcomings. In section 2 we elaborate the main requirements for a filter mechanism to significantly reduce the required effort to adapt trace fragments. Providing a filter mechanism is essential to empower an easy and flexible adaptation of fragments to continuously evolving information needs. The filter mechanism described in sections 3 and 4 allows a dynamical adaptation of trace fragments. Filters can be defined for a specific project phase as well as for particular trace fragments. A prototypical implementation of the filter mechanism was integrated into our process integrated environments and applied to small examples (section 5). Finally, we provide a conclusion of the achieved results and give an outlook on future work (section 6).
2 Requirements for a Model-based Filter Mechanism 2.1 Prevent Storage of Trace Information The filter mechanism has to ensure that certain trace information is not recorded in
240
Ralf Doemges, Klaus Pohl, Klaus Schreck
the trace repository. To avoid the re–programming of atomic fragments the filter mechanism should be able to partially block the output of an atomic fragment from being stored in the trace repository. For example, take the automated recording of process observation information by an atomic fragment. This fragment records all executed trace fragments together with the agent who executed them. Assume that due to contractual or organizational regulations the agents should not be recorded. A trace filter which could block the information about the agent of being stored in the trace repository avoids the re–programming of the atomic fragment. 2.2 Restrict Selection of Trace Strategies The filter mechanism must provide means to restrict the alternatives provided by a strategy selection fragment. Whenever an alternative of a strategy selection fragment should not be used, it should not be offered to the user. For example, a strategy selection fragment provides four alternatives to the user to justify the creation of a new product version: (1) recording a structured design decision, (2) stating the reasons for the change as informal text, (3) selecting an appropriate part of the project–contract, or (4) indicating the person creating the new version. If during the early project–phases justifications should only be given by informal text or by naming the responsible person the two other alternatives must be removed. 2.3 Prevent Execution of Trace Fragments The filter mechanism must allow to prevent the execution of a trace fragment. Whenever the entire output information of a trace fragment should not be recorded the execution of the fragment has to be prevented; especially if a trace fragments requires user interactions. For example, if design decisions should not be recorded during early project phases the execution of the trace fragment for recording the decision has to be prevented. 2.4 Enable Filter Definitions for the Overall Projects and Project Phases In addition to filter definitions for a single trace fragment the filter mechanism should provide means to define filters which affect all trace fragments executed in a project phase. For example, to capture no process observation information during the proposal phase whereas their recording is mandatory during the requirements engineering and design phases it should be possible to define a filter for an entire project phase. 2.5 Empower Nested Filter Definitions Trace fragments can be nested forming composed trace or strategy selection fragments. This requires propagation rules for filters defined for nested trace fragments. For example, assume that capturing process observation information is explicitly prohibited for a composed trace fragment. Consequently, none of the trace fragments used for defining the composed trace fragment should record process observation information, even if defined differently by filters of these trace fragments. 2.6 Enable Enforcement of Information Storage and Fragment Execution Within a nested trace fragment various information types can be blocked, alternative strategies can be restricted, and/or the execution of trace fragments can be prevented by defining the appropriate filters. For a composed trace or strategy selection fragment the filter mechanism must provide means to ensure the recording of a certain type of
A Filter-Mechanism for Method-Driven Trace Capture
241
information, the offering of particular alternative trace strategies, and/or the execution of specific trace fragments regardless of the filters defined for the fragments which are (constituent) parts of the composed fragments. For example, assume that the recording of automated dependency information is blocked by a filter F defined for a trace fragment tf. To assure the recording of this information in a composed trace fragment the project manager should be able to define a filter F’ which “replaces” F; i.e., which assures the recording although the nested filter F prohibits the recording.
3 Information, Strategy, and Method Filters To fulfill the requirements defined above three filter types are required: information filters which prevent already produced information from being stored in the repository (section 3.1), strategy filters which restrict the available trace capture strategies defined by a strategy selection fragment (section 3.2), and method filters which prevent trace fragments from being executed (section 3.3). filter
information filter
strategy filter
+1
has scope
abstract
abstract
method filter
block execution
+1
trace fragment
+2
+1
information type
block alternative
project phase
+1
abstract
composed of
alternative block information type
scope
+1
strategy selection fragment
atomic fragment
composed trace fragment
has output
Fig. 1. Filters types
These three filter types are defined as specialization of the concept filter (see figure 1). The scope defines if a filter is valid within a project phase or a trace fragment (see requirement 2.4). A filter can be related to more than one scope. If a filter should be applied in particular project phase(s), the filter is associated to the phase(s) by using the has scope association. If a filter is associated to all project phases, the filter is applied to the overall project. If a filter should only be valid for a particular trace fragment, the filter is related to the fragment using the has scope association. 3.1 Information Filters Information filters prevent information types from being stored persistently although the corresponding trace fragments are executed and their output information is produced (figure 2; requirement 2.1). The type of information to be blocked by a given information filter is defined using the association block information type defined between the class information type and the class information filter. The project phases and trace fragments to which an information filter should be applied are related to the information filter using the has scope association. An information filter can be used to avoid the recording of a particular information produced by an atomic fragment; but it can also be used to avoid the recording of a particular information during a project phase. An information filter can block more than one information type.
242
Ralf Doemges, Klaus Pohl, Klaus Schreck
Information Filter
Repository Decision: CreateRelShip "R"
Decision: CreateRelShip "R"
E1
1:n
R
1:n
E1
1:n
1:n
R
E2
E2 CreateCompleteRelShip27
NO process observation information
CreateRelShip32
CreateConnect56
Fig. 2. Information filters
Information filters are essential for influencing the information capture of atomic fragments. Information filters are very well suited for blocking process observation information which is typically recorded automatically. They are generally applicable for dependency information, especially if dependencies are created automatically. Information filters should not be used to filter supplementary product information, since these information types are usually created interactively. 3.2 Strategy Filters Strategy filters restrict the available alternatives of strategy selection fragments (figure 3; see requirement 2.2). By applying strategy filters to a strategy selection fragment only a subset of the defined alternatives is offered to the user. The alternative(s) which should not be offered are defined using the association block alternative defined between the class strategy filter and the association class alternative defined between the class strategy selection fragment and the class trace fragment (figure 1). A strategy filter can be related to more than one alternative fragment. Strategy Filter
Repository
select responsible agent
block strategy 3
select justification object
block strategy 1
2
select justification object
select design decision
select contract parts
3
1
select responsible agent
select responsible agent
2
2
select contract parts
select design decision
1
3
select justification object
Fig. 3. Strategy filters
The scope of the filter is defined by associating the appropriate project phases and/or trace fragments with the filter. If a strategy filter is associated to one or more project phases, the blocked alternatives are not offered within the defined phases; but are offered within other phases. If it is associated to a trace fragment, the blocked alternatives are not offered whenever the strategy selection fragment is executed within the trace fragment. 3.3 Method Filters Method filters prevent the execution of one (or more) atomic fragment, strategy selection fragment, or composed trace fragment (figure 4; see requirement 2.3). The trace fragments to be blocked are defined using the association block fragment defined
A Filter-Mechanism for Method-Driven Trace Capture
243
between the class method filter and the class trace fragment (figure 1). A method filter can block more than one method fragment. The scope of the filter is defined by using the has scope association to relate project phases and/or trace fragments with the filter. If a method filter is associated to one or more project phases, the blocked fragment is never executed within the defined phases; but it will be executed in other phases. If it is associated to a trace fragment, the blocked fragment is not executed within the associated fragment.
Process Filter
Repository Fragment X Fragment Y
Fragment X
PARTS OF Fragment X Fragment X’
Fragment Y
NO Fragment Y
Fig. 4. Method filters
Method filters should be used to adapt the interactive capture of traces, i.e., the recording of supplementary products and interactively created dependency information. They provide no means to change the internal control-flow of composed trace fragments. This can (only) be achieved by a manual adaptation of the fragment definition.
4 Nesting Information, Strategy, and Method Filters The trace fragments which define project-specific trace capture can be nested by defining composed trace or strategy selection fragments. Each of these nested trace fragments tf can be defined as the scope for a set of filters (denoted as filterset(tf)). As a consequence, the filters defined for the trace fragments are also nested. Consequently, we need to define propagation rules to determine the filters which have to be applied if a particular trace fragment is executed (section 4.1). filter
abstract
information filter
strategy filter
+1
has scope
mode: (permit, prevent)
method filter
block execution
abstract
+1
trace fragment
+2
+1
information type
block alternative
project phase
+1
abstract
composed of
alternative block information type
scope
+1
strategy selection fragment
atomic fragment
composed trace fragment
has output
Fig. 5. Filter modes
The (propagated) filter can not be used to enforce the recording of information types, that alternatives of a strategy selection fragment are offered, or the execution of particular trace fragments. For example, assume a filter associated to a trace fragment which is contained within a composed trace fragment. The filter defines to block
244
Ralf Doemges, Klaus Pohl, Klaus Schreck
the recording of a particular information type. If we want to ensure that the trace information will be recorded whenever the nested fragment is executed, we must provide means to define which information should be recorded during the execution of a composed trace fragment regardless of the filter definitions associated to its contained fragments. We therefore introduce a filter mode attribute (figure 5). Using this attribute the project manager is able to define explicitly that a filter prevents or permits (a) the defined information type to be recorded (information filter); (b) a certain alternative to be offered (strategy filter); (c) a trace fragment to be executed (method filter). Due to the filter mode the filters which have to be applied to a trace fragment may contradict each other. In section 4.2 we define these contradictions and describe how they can be resolved. Finally, we provide some rules towards a systematic application of trace filters (section 4.3). 4.1 Fragment Scopes and Valid Filter Sets For nested filters we determine the valid filter set of a trace fragment. The valid filter set consists of the filters to be applied to a fragment whenever the fragment is executed. We first define a containment–relationship between trace fragments: • A trace fragment tf directly contains another trace fragment tf’ (denoted as tf’ 2 tf) iff a.) tf is a composed method fragment and there exists a composed of link between tf and tf’. b.) tf is a strategy selection fragment and there exists an alternative link between tf and tf’. In other words, tf’ is used to define tf. • A trace fragment tf indirectly contains another trace fragment tf’ (denoted as 3 3 tf’ 2 tf) iff tf’ 2 tf or 9tf” with tf” 2 tf ^ tf’ 2 tf” Figure 6 depicts examples of this containment relationship for some trace fragments, 3 e.g., tf2 2 tf8 , tf1 2 tf2 , and tf1 2 tf8 . tf9
tf8 tf1
tf2
tf5 tf4
tf3
tf1
tf7
tf2 tf6
tf1
tf4 tf3
... ...
tf10
... Fig. 6. Containment and scopes of trace fragments
To determine a valid filter set we need to define the scope of a nested trace fragment tf. The scope of tf is a set of trace fragments which directly or indirectly contain each other. The scope of tf is defined relative to a particular trace fragment tf’ because tf may be (directly or indirectly) contained in more than one trace fragment. For example, a trace fragment to record design decisions can be used within a composed trace fragment which defines how to integrate changes into an existing specification as well as within a trace fragment which describes the creation of a new version of a specification. In figure 6 are tf1 2 tf2 and tf1 2 tf5 . Consequently, a trace fragment tf usually has more than one scope.
A Filter-Mechanism for Method-Driven Trace Capture
245
We define scope(tf; tf
2
0
)
=
(tfk0 ; . . . ; tfkn )
8i 2 f1; . . . ; ng,
and tfkn = tf 0 . If then scope(tf; tf 0 ) = (tf ) = (tfk0 ) = (tfkn ) = (tf 0 ). Figure 6 depicts some examples: scope(tf1 ,tf8 ) = (tf1 ,tf2 ,tf8 ) and scope(tf1,tf5 ) = (tf1 ,tf5 ) Based on filterset(tf) we define the valid filter set of a nested trace fragment tf (denoted as VFS(tf,tf’)) as the union of the filtersets of its corresponding scope: VFS(tf,tf’) = filterset(scope(tf,tf’)) filterset(tfki ). and filterset((tfk0 ; . . . ; tfkn )) = = = tfk0 =
with tf
tf
tfk0
tfkn
,
=
0
tfki 1 0 tf
tfki ;
S
i=0;...;n
VFS(tf,tf’) contains also the filters which are associated with the project phase in which tf (and the fragments of scope(tf,tf’)) is actually executed. By defining and using the scope of a trace fragment to define its valid filter set we have established the propagation rules to determine the filters which have to be applied when a particular trace fragment is executed. 4.2 Contradicting Information, Strategy, and Method Filter Definitions A valid filter set may contain filters having contradicting filter modes. For example, if the recording of design decisions is enforced on the level of a composed trace fragment (i.e., the filter mode is set to permit) but prohibited on the level of trace fragments which are contained in the composed trace fragment it is not clear which filter should be applied. We thus need to define those contradictions and how to resolve them. In the following we denote filterobjects(F) of an information, strategy, or method filter F as the set of all information types, alternatives, or trace fragments to be permitted or prevented by F. Contradictions between two filters of the same type are defined and resolved as follows: • Let IF and IF’ be two information filters, filterobjects(IF) = {it1 ,...,itn }, n ≥ 1, and filterobjects(IF’) = {it’1 ,...,it’m }, m ≥ 1. The filter definitions of IF and IF’ are contradictory iff the filter mode of IF is permit, the filter mode of IF’ is prevent (or vice versa), and {it1 ,...,itn } \ {it’1 ,...,it’m } = fitk1 ; . . . ; itkl g, l ≥ 1. If IF, IF’ 2 VFS(tf,tf’) are two contradictory information filters, scope(tf, tf’) = {tf1 ,...,tfm }, IF’ 2 filterset(tfi ), IF 2 filterset(tfj ) and i > j, the filter definitions of IF’ for fitk1 ; . . . ; itkl g replace the filter definitions of IF for fitk1 ; . . . ; itkl g by removing them from VFS(tf,tf’) (figure 7 a.)). a.)
b.)
IF’ mode: prevent
F’ mode: prevent
it’p1,...,it’pr ,itk1,...,itkl
its1,...,itst ,itk1,...,itkl
...
tfj
and
F’
es F mode: permit tfsF1 ,...,tfsFt ,tfkF’ ,...,tfkF’ 1 l
...
...
pl ac
tfi
re
re
IF mode: permit
pl ac
es
tfpF’ ,...,tfpF’ ,tfkF’ ,...,tfkF’ 1 r 1 l
...
tfj
...
tfi
...
Fig. 7 Contradictory filter definitions
• Let
F
be
two
strategy
filters
(or
two
method
filters),
246
Ralf Doemges, Klaus Pohl, Klaus Schreck
filterobjects(F) = {tfF 1 ,...,tfF n }, n ≥ 1, and filterobjects(F’) = {tfF’ 1 ,...,tfF’ m }, m ≥ 1. The filter definitions of F and F’ are contradictory iff the filter mode of F is permit, the filter 8mode of F’ is9 prevent (or vice versa), and {tfF 1 ,...,tfF n } \ {tfF’ 1 ,...,tfF’ m } = tfkF1 ; . . . ; tfkFl , l ≥ 1. If F, F’ 2 VFS(tf,tf’) are two contradictory strategy filters (or method filters), scope(tf,tf’) = {tf1 ,...,tfp }, 8F’ 2 filterset(tf i ), F 2 filterset(tfj ) and i > j, the F F 9 replace the filter definitions of F for tfk ; . . . ; tfk filter definitions of F’ for 1 l 8 F F 9 by removing them from VFS(tf,tf’) (figure 7 b.)). tfk ; . . . ; tfk 1 l If tfi = tfj (i.e., i = j) the same fragment is associated with contradicting filter definitions. In this case the project manager has to decide which of the filter definitions should be replaced. Two filters of different types are contradictory iff IF is an information filter, F is a strategy filter (or method filter), filterobjects(IF) = {it1 ,...,itn }, filterobjects(F) = {tf’1 ,...,tf’m’ }, the filter mode of IF is permit, the filter mode of 3 F is prevent, and there exists an atomic fragment af 2 tf’l , l 2 {1,...,m’} which produces fitj1 ; . . . ; itjk g fit1; . . . ; itn g. If IF, F 2 VFS(tf,tf’) are two contradicting filters, where IF is an information filter and F is a strategy filter (or method filter), scope(tf,tf’) = {tf1 ,...,tfm }, IF 2 filterset(tfi ), F 2 filterset(tfj ) and i ≥ j the contradiction can not be resolved automatically but the project manager has to decide how to resolve the contradiction. Thereto s/he needs to 3 determine the trace fragment tf’l which contains the affected atomic fragment af 2 tf’l and has to figure out how to adapt the filter definitions of IF and F, e.g., by preventing the execution of all trace fragments which are contained within tf’l except af. There will be no contradictions between the filter definitions of a method filter MF and a strategy filter SF. We demand that alternatives of strategy selection fragments can only be prevented by strategy filters (see section 4.3) and strategy filters are only able to restrict the alternatives of a strategy selection fragment. Even if a method filter defines to prevent the execution of an alternative it is still offered to the user. If the filter definitions associated with the project phase in which a trace fragment tf (and the fragments of scope(tf,tf’)) is actually executed contradict with VFS(tf,tf’) the filter modes of the project phase generally replace the filter modes of the trace fragments. The above definitions can be used to analyze the filters defined for the trace fragments. Contradicting filters can thus be detected before the trace fragments are actually applied. The project manager can resolve the contradictions before the fragments are executed during a project. 4.3 Rules for applying Filters Based on our experience we provide some rules for applying information, strategy, and method filters: Apply filters not to product information: Product information should never be affected by filters. Product information is the main output of the development process. Hence, it makes no sense to block their recording. For example blocking product information during the development of a Entity–Relationship model would lead to an incomplete and inconsistent model. Filters should only affect the recording of supplementary product, process observation, and dependency information. If a change in product information is required (e.g., define inheritance (links) in En-
A Filter-Mechanism for Method-Driven Trace Capture
247
tity–Relationship–diagrams) new fragments have to be introduced and/or existing fragments have to be adapted. This definition and/or re–definition of a method is not within the scope of a filter mechanism. Apply information filters only to automated trace fragments: If the information of interactive trace fragments is blocked by information filters it is very likely that users reject to enter the information next time. This might lead to the rejection of the entire filter–based approach for capturing traces. Information filters should thus never be used to block interactively entered information. Apply method or strategy filters when complete output information is blocked: A fragment whose complete output is blocked by (nested) information filters should not be executed. Instead, a method filter should be defined to prevent the execution, or if the fragment is an alternative of a strategy selection fragment, an appropriate strategy filter should be defined. Apply method filters when all alternatives of a strategy selection fragment are prevented: If the entire set of alternatives of a strategy selection fragment is prevented by (nested) strategy filters, the fragment should not be executed. Instead of defining strategy filters which block all alternatives, a method filter should be defined to prevent the execution of the strategy selection fragment. Check effects on composed trace fragments: If any kind of filters prevent the storage of information or the execution of a trace fragment within a composed trace fragment, the project manager must check if the blocking of the information (or the fragment) does not lead to a “deadlock”. In other words, s/he must assure that a composed trace fragment could be executed although a trace fragment is blocked and/or information is not recorded. In the case of a detected deadlock s/he must change the control flow of the composed trace fragment. Do not apply method filters to block alternatives of strategy selection fragments: Method filters should not be misused as strategy filters, i.e., they should not be used to block an alternative of a strategy selection fragment. By defining a strategy filter, the alternative is not offered to the user, whereas in the case of a method filter, the alternative is offered to the user. The user can choose the alternative, but the chosen alternative will not be executed. Together with the scope and contradictions defined in sections 4.1 and 4.2 the rules provide the basis for developing an environment which supports the project manager in defining consistent trace filters of any type.
5 Model–based Filtering: An Example We illustrate our model-based filter mechanism using a small example. The composed trace fragment integrate change request guides the application engineer during the integration of changes (figure 8). The application engineer is first reminded to justify the changes. The strategy selection fragment select justification object defines three alternative strategies for the justification: (1) to select appropriate parts of a contract; (2) select the stakeholder who initiated the change; or (3) to select a specific design decision. A process observation fragment automatically records the execution of the strategy selection fragment and the chosen alternative. During the integration of changes an automated dependency step relates the object representing the justification with the modified and/or created specification parts.
Fig. 8. Composed trace fragment integrate change requests (simplified).
The fragment described above is reused for the proposal phase of another project. The project management decides that in this project it is sufficient to justify the change by stating the responsible stakeholder. In other words, the two other alternatives of the strategy selection fragment should not be offered. Since two of the three alternatives of the strategy selection fragments are blocked, the chosen alternative needs not to be recorded by the process observation step. Moreover the project manager decides that no dependencies should be created between the stakeholder initiated the changes and the modified or created specification parts. We use our filter mechanism to adapt the method fragment integrate change request according to the new requirements of the project manager. We define • one strategy filter which blocks the alternatives select contract parts and select design decisions of the strategy selection fragment select justification object; • one method filter which prevents the execution of the atomic fragment create dependency; instead of associating an information filter with the atomic fragment to block its entire output; • one information filter which blocks the recording of the information about the chosen alternatives. This filter is associated to the record strategy selection fragment. composed trace fragment: integrate change request strategy selection fragment: select justification object
S E L E C T
composed trace fragment
select contract parts
atomic fragment
create dependency
contract statements
records based_on dependency link
atomic fragment
select responsible agent atomic fragment
select design decision
agent
design decision atomic fragment
record strategy selection
justification object
composed trace fragment
specification objects
change specification
select justification object/ chosen alternative
Fig. 9. Adapted trace fragment integrate justified change (simplified)
The application of these filters leads to the trace fragment(s) depicted in figure 9. The parts of the trace fragment which are not executed, i.e., prevented by the filters, are depicted in grey. This changes could be achieved without any re–modeling of the composed trace and strategy selection fragments and without any reprogramming of the atomic method fragments.
A Filter-Mechanism for Method-Driven Trace Capture
249
6 Conclusion and Future Work Our approach to method–driven trace capture [Pohl et al., 1997] enables the definition of project–specific trace information and trace capture strategies. Based on this definitions the user is guided in capturing the required project-specific trace information. Originating from its application in case studies two main shortcomings of the approach were recognized: adapting trace fragments to varying traces during a project required a significant effort for (re–)modeling and (re–)programming; managing and maintaining the trace fragments became almost impossible due to redundant parts of the introduced fragments and a rapidly increasing amount of trace fragments. The filter–mechanism presented in this paper avoids the two shortcomings. Based on a set of requirements for trace filters we have defined three types of filters: • information filters block certain information types from being stored in the repository; • strategy filters restrict the alternative trace strategies offered to the user; • method filters prevent a trace fragment from being executed. A filter can be defined for particular project phases or specific trace fragments. The filter definitions influence the recording of the traces during a project phase and/or during the execution of a trace fragment. To enforce the recording of certain information we have defined two filter modes: prevent and permit. We defined propagation rules for nested filters to determine all filters to be applied for a trace fragment whenever it is executed and specified how to resolve resulting contradictory filter definitions. To support the systematic definition of filters we provided a set of rules for their application. The filter mechanism was validated by integrating it into the TECHMOD and PRO–ART 2.0 environments and by applying it to small examples. Early experience confirms that trace filters significantly reduce the necessary effort to adapt trace fragments and facilitates the management and maintenance of the method base. The development of tool support for the definition and application of filters will be focus of our future work. Such support should employ the defined rules for applying filters and provide mechanisms to check the effects of filters on the trace fragment definitions.
Acknowledgments This work was supported by the DFG-Projekt “Prozeß-Integration von ModellierungsArbeitspl¨atzen”, the ESPRIT Long Term Research Project 21903 CREWS (Cooperative Requirements Engineering With Scenarios), and the DAAD/ACLS/NSF program “Technische und empirische Grundlagen der Requirements Traceability: Modelle und Mechanismen”. The authors are grateful to their colleagues P. Haumer, M. Jarke, K. Weidenhaupt, and S. Zlatintsis for many fruitful discussions and contributions.
References [Ascent Logic Corporation, 1994] Ascent Logic Corporation. RDD–100 Marketing Brochure, 1994. [Br¨ohl and Dr¨oschel, 1993] A.P. Br¨ohl and W. Dr¨oschel. Das V–Modell. Oldenbourg Verlag, 1993.
250
Ralf Doemges, Klaus Pohl, Klaus Schreck
[Collery, 1988] A. Collery. Traceability, the New Strategic Challenge for Companies, its Tool, Character Reading in Industrial Circles. In Proc. of the 19th Intl. Symposium on Automotive Technology and Automation, with Particular Reference to Cell Control and Quality Management Systems for the Manufacturing Industries, volume 1, pages 251–260, Monte Carlo, Monaco, October 1988. Allied Automation. [Conklin and Begeman, 1988] J. Conklin and M.J. Begeman. gIBIS: A Hypertext Tool for Exploratory Policy Discussion. ACM Transactions on Office Information Systems, 6(4):303– 331, 1988. [DoD-2167A, 1988] DoD-2167A. Military Standard: Defense System Software Development. 1988. U.S. Dept. of Defense. [D¨omges et al., 1996] R. D¨omges, K. Pohl, M. Jarke, B. Lohmann, and W. Marquardt. PRO– ART/CE — An Environment for Managing Chemical Process Simulation Models. In Proc. of the 10th Europ. Simulation Multiconference, pages 1012–1017, Budapest, Hungary, June 1996. [Gotel, 1996] O. Gotel. Contribution Structures for Requirements Engineering. PhD thesis, Imperial College of Science, Technology, and Medicine, London, England, 1996. [IEE, 1991] IEE. Proceedings of the IEE Colloquium on Tools, Techniques for Maintaining Traceability During Design. London, England, December 1991. [ISO, 1991] ISO. ISO9000–3: Quality Management and Quality Assurance Standards. International Institute for Standardization, Genf, Switzerland, 1991. [Jarke et al., 1994] M. Jarke, K. Pohl, C. Rolland, and J.-R. Schmitt. Experience-Based Method Evaluation and Improvement: A Process Modeling Approach. In IFIP WG 8.1 Conference CRIS ’94, Maastricht, The Netherlands, 1994. [Kaindl, 1993] H. Kaindl. The Missing Link in Requirements Engineering. ACM SIGSOFT Software Engineering Notes, 19(2):30–39, 1993. [Marconi Systems Technology, 1996] Marconi Systems Technology. RTM (Requirements & Traceability Management) – Marketing Information, 1996. [Paulk et al., 1993] M. Paulk, B. Curtis, M. Chrissis, and C. Weber. Capability Maturity Model for Software: Version 1.1. Technical Report SEI-93–TR-24, Software Engineering Institute, Carnegie Mellon University, Pittsburg, Pennsylvenia, USA, February 1993. [Pinheiro and Goguen, 1996] F.A.C. Pinheiro and J.A. Goguen. An Object–Oriented Tool for Tracing Requirements. IEEE Software, pages 52–64, March 1996. [Pohl and D¨omges, 1997] K. Pohl and R. D¨omges. An Environment for Model–Based Trace Capture. In Proc. of the Intl. Conf. on Software Engineering and Knowledge Engineering, Madrid, Spain, June 1997. [Pohl and Weidenhaupt, 1997] K. Pohl and K. Weidenhaupt. A Contextual Approach for Process–Integrated Tools. In Proc. of the 6th Europ. Software Engineering Conference, Z¨urich, Switzerland, September 1997. [Pohl et al., 1997] K. Pohl, R. D¨omges, and M. Jarke. Towards Method–Driven Trace Capture. In Proc. of the 9th Intl. Conf. on Advanced Information Systems Engineering, Barcelona, Spain, June 1997. [Pohl, 1996a] K. Pohl. PRO–ART: Enabling Requirements Pre–Traceability. In Proc. of the 2nd Intl. Conf. on Requirements Engineering, Colorado-Springs, Colorado, USA, April 1996. [Pohl, 1996b] K. Pohl. Process Centered Requirements Engineering. RSP by J. Wiley & Sons Ltd., England, 1996. [Quality Systems & Software, 1996] Quality Systems & Software. DOORS (Dynamic Object Oriented Requirements System) – Marketing Information, 1996. [Ramesh et al., 1996] B. Ramesh, C. Stubbs, T. Powers, and M. Edwards. Implementing Requirements Traceability: A Case Study. Annals of Software Engineering, 9:1–19, 1996. [Rolland and Grosz, 1994] C. Rolland and G. Grosz. A General Framework for Describing the Requirements Engineering Process. In Proc. of the Intl. Conf. on Systems, Man, and Cybernetics, San Antonio, Texas, USA, October 1994. IEEE Computer Society Press. [TD Technologies, Inc., 1996] TD Technologies, Inc. SLATE (System Level Automation Tool for Engineers) – Marketing Information, 1996. [Tilbury, 1989] A.J.M. Tilbury. Enabling software traceability. In Proc. of the IEE Colloquium on The Application of Computer Aided Software Engineering Tools, pages 7/1–7/4, London, England, February 1989. [Yu and Mylopoulos, 1994] E. Yu and J. Mylopoulos. Using Goals, Rules, and Methods to Support Reasoning in Business Process Reengineering. In Proc. of the 27th Hawaii Intl. Conf. on System Sciences, volume IV, pages 234–243, Maui, Hawaii, USA, January 1994.
Subject-Based Organization of the Information Space in Multi-Database Networks Michael P. Papazoglou1 and Steven Milliner2
2
1 Tilburg University, INFOLAB, P.O. Box 90153, 5000 LE Tilburg, The Netherlands [email protected] Queensland University of Technology, School of Information Systems, GPO Box 2434, Brisbane QLD 4001, Australia [email protected]
Abstract. Rapid growth in the volume of network-available data, com-
plexity, diversity and terminological uctuations, at di erent data sources, render network-accessible information increasingly dicult to achieve. The situation is particularly cumbersome for users of multi-database systems who are expected to have prior detailed knowledge of the de nition and uses of the information content in these systems. This paper presents a conceptual organization of the information space across collections of component systems in multi-databases that provides serendipity, exploration and contextualization support so that users can achieve logical connections between concepts they are familiar with and schema terms employed in multi-database systems. Large-scale searching for multi-database schema information is guided by a combination of lexical, structural and semantic aspects of schema terms in order to reveal more meaning both about the contents of a requested information term and about its placement within the distributed information space.
1
Introduction
The dramatic growth in global interconnectivity has placed vast amounts of data within easy reach. At the same time it has made on-demand access to widelydistributed data a natural expectation for a variety of users. A limiting factor however, is the diculty in providing coherent access and correlation of data that originate from diverse widely-distributed data sources. This is an involved process not only due to the sheer volume of information available, but also because of heterogeneity in naming conventions, meanings and modes of data usage. Differences in data descriptions, abstraction levels, and precise meanings of terms being used in disparate data sources do not yield well at all to automation. These problems are compounded by di erences in user perceptions and interpretations, and variations that may occur at autonomous database sites over time. Users are thus presented with the problem of gaining adequate knowledge of a potentially huge, complex dynamic system, in order to access and combine information in B. Pernici and C. Thanos (Eds.): CAiSE’98, LNCS 1413, pp. 251-272, 1998. Springer-Verlag Berlin Heidelberg 1998
252
M.P. Papazoglou and S. Milliner
a coherent and logical manner. Yet multi-database systems demand from users prior detailed knowledge of the de nition and uses of their underlying data [24]. This expectation is quite unreasonable in large distributed systems. The focus in multi-database systems is on query processing techniques and not on how to discover where the actual schema elements in the component systems reside. No particular attention is paid to how schema items are structured, what they mean and how they are related to each across component database schemas. The user's perception of the information content in networked databases is that of a vast space of information in a large at, disorganized set of database servers. In contrast to this, our approach to searches for widely distributed information concentrates on providing a dynamic, incremental and scalable logical organization of component database sources, and search tools that are guided by this organization. We view user interaction with a multi-database space as comprising two major phases, the:
schema information discovery phase where users systematically explore the multi-database space to locate potentially useful databases, and the
distributed query/transaction phase where the requested data sets are retrieved from the candidate databases.
We consider the development of a methodical, scalable search process critical to the successful delivery of information from networked database systems. Hence, in order to provide users with tools for the logical exploration of distributed information sources a four step process, termed information elicitation is introduced and includes: (i) Determining the information needs of users by means of di erent term suggestions; (ii) Locating candidate database sources that address these needs; (iii) Selecting schema items of interest from these sources; and nally, (iv) Understanding the structure, terminology and patterns of use of these schema items which can subsequently be used for querying/transaction purposes. The very nature of this process suggests that we should provide facilities to landscape the information available in large multi-database networks and allow the users to deal with a controlled amount of material at a time, while providing more detail as the user looks more closely. To support the process of information elicitation while overcoming the complexity of wide-area information delivery and management, we cannot rely on a collection of indexes which simply contain schema information exported by individual database sources. A more structured and pro-active approach to searching is required. The precursor of such an advanced search approach assumes that we are in a position to impose some logical organization of the distributed information space in such a way that potential relationships between the component database systems in the network can be explored. In addition, to maintain scalability, this must be achieved through a decentralized mechanism which does not proceed via a one step resolution and merging of system information into a single static monolithic structure. These and related issues are addressed herein.
Subject-Based Organization of the Information Space in Multi-database Networks
253
This paper is organized as follows. Section 2 presents related work, while section 3 discusses a logical organization for the semantic cross correlation of metadata information from component databases in a multi-database system. Section 4 presents clustering techniques, while section 5 outlines navigation and querying mechanisms. Finally, section 6 presents our conclusions and future work. This work is an extension and elaboration of some early ideas outlined in [14] and [15]. In [14] we concentrated on the organization of physical data sharing in large database networks, and described how physical data sharing ties in with a pre-cursor of the conceptual organization of the information space presented in this paper. In [15] we described IR techniques and algorithms used for the physical clustering of databases. In this paper we concentrate on the details of logical database formation, according to subject, based on a common terminology context and present navigation and querying techniques.
2 Finding Information: An Overview In this section a number of techniques from di erent elds for locating information are discussed. Web-based Resource Discovery
The use of the World Wide Web (WWW) has led to the development of a variety of search engines which attempt to locate a large number of WWW documents by indexing large portions of the Web. These tools recursively enumerate hypertext links starting with some known documents. We can classify search engines into two broad categories: centralized index and content-based search engines. Centralized index search engines such as Lycos [11], Web Crawler [19] are manual indexing schemes that rely on techniques which \crawl" the network compiling a master index. The index can then be used as a basis for keyword searches. These systems are not scalable because they use a global indexing strategy, i.e., they attempt to build one central database that indexes everything. Such indexing schemes are rather primitive as they cannot focus their content on a speci c topic (or categorize documents for that matter): as the scope of the index coverage expands, indexes succumb to problems of large retrieval sets and problems of cross disciplinary semantic drift. Some of the above limitations are addressed by content-based search engines such as the Content Routing System [23] and Harvest [2]. These systems generate summarized descriptions (content labels) of the contents of information servers. The Content Routing System creates and maintains indexes of widely distributed sites. In this distributed information retrieval system a collection of documents is described by means of a content label which in turn can be treated as a document and can be included in another collection. Content labels help users explore large information spaces. However, document collections and their labels are con ned to the context of their underlying information servers. Recently, this idea has been extended in the HyPersuit system [26] by generalizing collections so that they may span documents from various servers.
254
M.P. Papazoglou and S. Milliner
The Harvest information discovery and access system [2] provides an integrated set of tools for gathering information from diverse Internet servers. It builds topic-speci c content indexes (summaries from distributed information), provides ecient search mechanisms, and caches objects as they are retrieved across the Internet. Each local search engine builds a specialized directory for a certain domain of documents. Federated search engines scan those directories and form federated directories which aggregate documents according to applicationspeci c needs.
Subject Gateways
A subject gateway, in network-based information access, is de ned as a facility that allows easier access to network-based information resources in a de ned subject area [9]. Subject gateways o er a system consisting of a database and various indexes that can be searched through a Web-based interface. Each entry in the database contains information about a network-based resource, such as a Web page, Web site or document. Advanced gateways provide facilities for enhanced searching. For example the Social Science Information Gateway (SOSIG) [25], incorporates a thesaurus containing social science terminology. This gives users the option of generating alternative terms/keywords with which to search the resource catalog. Another example of an advanced subject gateway is the Organization of Medical Networked Information (OMNI) [16] which allows users to access medical and health-related information. OMNI also facilitates searches across other databases of resources such as databases of dental resources. The key di erence between subject gateways and the popular Web search engines, e.g., Alta Vista, lies in the way that these perform indexing. Alta Vista indexes individual pages and not resources. For example, a large document consisting of many Web pages hyperlinked together via a table of contents would be indexed in a random fashion. In contrast this subject gateways, such as OMNI, index at the resource level, thus, describing a resource composed of many Web pages in a much more coherent fashion.
Multi-Database Systems
Multi-database (or federated) systems have as their aim the ability to access multiple autonomous databases through querying. The emphasis is on integration and sharing of distributed information and not on information discovery. A particular database may choose to export parts of its schema which are registered in a federal dictionary. A requesting database consults the federal dictionary for existing databases and then imports schema elements that it requires. While this approach might be appealing for a small number of interconnected databases it is clearly not scalable. Locating the right information in a large unstructured network of data dictionaries is extremely cumbersome, has limited potential for success and, more importantly, is error prone as it does not deal with terminology nuances.
Subject-Based Organization of the Information Space in Multi-database Networks
255
More recently several research activities in the area have concentrated on the issue of creating semantically enhanced federated database dictionaries [3], [1], [12], [4]. Construction of conceptual ontologies on the basis of domain-speci c terminologies and formalisms that can be mapped to description logics are also discussed in [8]. Some of the issues relating to the identi cation of semantically related information can be found in [3], where the authors describe an approach that relies on an abstract global data structure to match user terms to the semantically closest available system terms. Concepts grounded on a common dictionary are de ned in a domain and schema elements from component databases are manually mapped to these concepts. More recently, a di erent approach is taken by [7] where a domain-speci c classi cation scheme is built incrementally by considering one schema at a time and mapping its elements in a concept hierarchy. However, both these approaches tend to centralize the search within a single logical index thereby defeating scalability by introducing performance limitations for large networks.
3 System Organization In order to improve ecient searching/elicitation of schema information in large multi-database networks, the rst task is to partition the multi-database information space into distinct subject (domain-speci c) categories meaningful to database users. Categorization and subject classi cation are common practices in library and information sciences, e.g., the INSPEC indexing and abstracting service covering most of the research literature in Computer Science and Electrical Engineering [22]. Domain-speci c partitioning organizes databases in logical clusters and makes searches more directed, meaningful and ecient. In addition, a subject directory created as a result of domain-speci c database categorization can also provide subject-speci c searches and useful browsable organization of inter-component database schema information. There are three basic principles that a system must address to allow for scalable information elicitation. Firstly, an organization of raw data must be introduced for the discovery of data inter-relationships. Topic classi cation schemes for this purpose as they summarize related information subspaces together. Secondly, this organizational structure must itself be scalable { that is: interactions with it must be scalable, and maintenance of it must be scalable. Thirdly, users must be presented with a collection of tools (lexicographic, and user friendly graphical interfaces) which allows for easy exploration and interpretation of the information contents of the system. In the following, we address these issues in the context of a logical topic-based architecture for multi-databases. 3.1
Subject-based Database Clustering
Our approach to information elicitation in large database networks relies on logically partitioning the multi-database schema information space into distinct subject (topic) categories meaningful to users. This occurs by creating logical
256
M.P. Papazoglou and S. Milliner
objects called Generic Concepts (GCs) to achieve explicit semantic clustering of associated component database schema elements. Database-content clustering automatically computes sets of related component databases { via their exported meta-data terms { and associates them with an appropriate generic concept, see Figure 1. Generic concepts essentially represent centroids of the inter{component database schema information space { around which databases cluster { and are engineered to describe a particular domain (generic concepts were termed \Global Concepts" in previous work [14]).
WWW−based user interface MULTIDABASE−NETWORK CONTENTS: EDUCATION − DOMAIN. − EDUCATION & TRAINING − SCIENTIFIC PUBLICATIONS − GOVERNMENT DEPTS − ....
Partitioning a multi-database information space into generic concepts.
To participate in GC-structured database network, a component database must export part of its meta-data to the other databases in the network. This
Subject-Based Organization of the Information Space in Multi-database Networks
257
means that the component database administrator must specify which part of the database meta-data can be made available for sharing with other database systems in the network. We refer to these meta-data as the exported meta-data. Figure 1 shows a sample database, called the Universal Accreditation Company database, along with a partial representation of its meta-data. Although metadata contain also physical de nitions such as de nitions of views, ownership, authorization privileges, indexes and access patterns, these (except for authorization privileges) are not important for inclusion in the GC level. A GC organized multi-database schema information space can be viewed as a Web-space that encompasses collections of exported meta-data. A GC organized multi-database schema information space partitions component databases into topically-coherent groups, and presents descriptive term summaries and an extended vocabulary of terms for searching and querying the vastly distributed information space of the component databases that underly it. Databases in this network may connect to more than one GCs if they strongly relate to their content. To circumvent terminology uctuations we provide a standard vocabulary for interacting with the GCs. In this way we create a concept space (information sub-space) for a speci c topic category. The concept space constitutes a type of summarization or synoptic topic knowledge regarding a particular domain, e.g., education and training, publications, government tertiary-related departments, etc, and is stored in a GC, see Figure 1. This clustering mechanism results in grouping exported meta-data elements from diverse databases that share important common properties onto a generic concept, associating these properties with the GC representation, and regarding the GC as an atomic unit. A GC is thus a form of a logical object whose purpose is to cross-correlate, collate, and summarize the meta-data descriptions of semantically related network-accessible data. This scheme provides an appropriate frame of reference for both component database schema term indexing and user instigated searches. With this scheme navigation can be considered as browsing through databases exclusively at a topic-level i.e., from topic area to topic area such as from educational training, to publications, government departments and so on. To put the organization of a concept space into perspective, we consider the case of a domain based on educational information provided by a large number of interconnected databases as shown in Figure 1. This gure also illustrates how a component database (Accreditation) { which provides information about accreditation of courses and cross-institutional subjects, various private/public educational training information and other similar or related data { is connected to the GC network. In its original form the Accreditation database, maintains information only on education service providers, their courses, accreditation committee members, accreditation processes and related information. Figure 1 shows the Accreditation database along with a partial representation of its associated meta-data and schema. It also illustrates how this component database may become part of a larger network by establishing weighted links to GCs implementing related areas of interest. Consequently, the Accreditation database is not only able to source appropriate information on its subject matter but also
258
M.P. Papazoglou and S. Milliner
to provide matching information about enrollment programs, training schemes, government programs, research activities and publication data. By linking to a certain GC, databases agree to associate with each other and thus inter{component database organization is achieved implicitly. In addition, GCs are interconnected by weighted links (called content links) to make the searches more directed and meaningful, see Figure 1. Each of the component databases may also link less strongly (e.g., 7/10) to other GCs which have their own associated cluster of database nodes. Presently, the degree of relatedness between GCs is decided by database administrators. Accordingly, a single database, e.g., Universal Accreditation Company, may be simultaneously involved in several clusters of databases (information sub-spaces) to varying degrees, as dictated by the weights of its content links to the various GCs. The resulting GC structure forms a massive dynamic network, resembling a cluster{based associative network (a variant of semantic networks that uses numerically weighted similarity links). Overall a networked information system may be viewed in terms of three logical levels. The bottom level (Figure 1) corresponds to the schemas of the component databases. The middle level represents exported meta{data for the database schemas. The top most level corresponds to the concept space (GC) level. This level contains abstract dynamic objects which implement the clustering of related portions of the underlying component meta-data and materialize the GCs in an object-oriented form. Figure 1 illustrates that there is a one-to-one correspondence between database schemas and their meta-data representations, while an entire collection of exported meta-data corresponds to a single concept-space. This three-tier architecture is the key ingredient to information elicitation in distributed, scalable systems. It provides the ability to describe varying levels of aggregated database sources and the granularity of the information components, i.e., exported meta-data terms, that comprise them. It generates a semantic hierarchy for database schema terms in layers of increasing semantic detail (i.e., from the name of a term contained in a database schema, to its structural description in the meta-data level, and nally to the concept space level where the entire semantic context { as well as patterns of usage { of a term can be found). Searches always target the richest semantic level, viz. GC level, and percolate to the schema level in order to provide access to the contents of a component database, see section 5. This type of content-based clustering of the searchable information space provides convenient abstraction demarcators for both the users and the system to make their searches more targeted, scalable and e ective. This methodology results in a simpli cation of the way that information pertaining to a large number of interrelated database schemas can be viewed and more importantly it achieves a form of global visibility [17]. Although GCs provide synoptic information about their underlying database clusters, they do not require integration of the data sources. This approach comes in strong contrast with approaches to semantic interoperability based on explicit integration of conceptual schemas on the basis of semantic lexica [3], [4]. The advantage of forming conceptual
Subject-Based Organization of the Information Space in Multi-database Networks
259
database clusters is that searches are goal{driven3 and the number of potential inter{database interactions is restricted substantially as it facilitates the distribution and balancing of resources via appropriate allocation to the various database partitions. 3.2
Generic Concept Characteristics
Individual GCs are useful for browsing and searching large database collections because they organize the information space. For example, the Education and Training Providers concept space provides a common terminology basis upon which database nodes dealing with enrollments, courses, training, accreditation, etc, (see Figure 1), achieve knowledge of each others information content. A GC is a de nitional or schematic construct: it corresponds to a class hierarchy depicting all terms within the topic sampled by the GC. The GC structure is illustrated in Figure 2. This gure shows that each GC is characterized by its name and the context of its terms (term hierarchy and term descriptions) for each speci c topic. Terms within a GC are shown to have a distinct meaning (sense) and context. This concept space consists of abstract descriptions of terms in the domain, term senses, relationships between these terms, composition of terms, terminology descriptions, hypernym, hyponym, antonyms-of, part-of, member-of (and the inverses), pertains-to relations, contextual usage (narrative descriptions), a list of keywords, and other domain speci c information, that apply to the entire collection of members of a GC, Figure 2. Hence, the GC structure is akin to an associative thesaurus and on-line lexicon (created automatically for each topic category). Thesaurus-assisted explanations created for each subject-based abstraction (GC-based information subspace) serve as a means of disambiguating term meanings, and addressing terminology and semantic problems. Therefore, the GC assists the user to nd where a speci c term that the user has requested lies in its conceptual space and allows users to pick other term descriptions semantically related to the requested term. Operations on a GC object include mapping services which map GC provided terms to semantically related terms in the component databases. They also include summarization services which summarize the exported meta-data from component databases to implement a GC. Summarization services aggregate networks of exported meta-data terms (one per component database). This mechanism is described in a later section. An example of the GUI for some of the the terms included in the educational GC is given in Figure 3. Here, we assume that a user who searches the entries in the educational GC is interested in the term course and wishes to gain more insight into its semantic context. The rst step after entering the term is to choose the senses from the list the GC lexicographic substrate provides. The sense number returned is then associated with the term (as is the case with all other words in the term description). For example, Figure 3 shows that the 3 A goal{driven search accepts a high{level request indicating what a user requires and is responsible for deciding where and how to satisfy it.
260
M.P. Papazoglou and S. Milliner
GC NAME: TERM STRUCTURE/COMPOSITION Term Hierarchy: (specialized/generalized terms) Term Senses: (meaning) Synonyms: (similar terms) Antonyms: (opposite terms) Hyponyms: (subordinate terms) Hepernyms: (superordinate terms) Meronyms: (part terms) Term Description: (narrative) CONTENT−LINKS TO OTHER GCs IN THE NETWORK OPERATIONS: (not user accessible) − mapping services − summarization services Fig. 2. Generic concept structure.
term course has eight senses (meanings), but once the domain of discourse is limited to study (education), then only one of the eight can occur. Figure 4 which is an expansion of the speci c term chosen, shows how the GC provides the necessary information needed for the contextual representation, i.e., meaning, of a speci c term. Other factors such as the context of usage (not shown here due to space limitations) can be combined with its contextual representation to restrict the search space. Thus the user gets a complete picture regarding the semantic context of this and associated terms (see Figure 4) and is free to pick up a desired term(s) which would eventually lead him/her to candidate component data sources. Term entries in this GUI are mapped by means of the mapping services of a GC to the relevant schema terms found in component databases (in the same GC). Information contained in the GCs is stored in an information-repository that resides at a concept server associated with and accessible by the databases clustered around a speci c conceptual information space (GC), see Figure 1. The concept server implements an individual GC, performing abstraction and summarization operations on its underlying meta-data. This information-repository contains thus a rich domain model that enables describing properties of the database sources clustered around a GC.
4 Representation and Clustering of Schema Meta-data In the following we describe a general methodology that aids in clustering databases and creating their corresponding generic concepts. Key criteria that have guided this methodology are: scalability, design simplicity and easy to use structuring mechanisms.
Subject-Based Organization of the Information Space in Multi-database Networks
Fig. 3.
4.1
261
Choosing the meaning of the term course.
Describing the Meta-Data Content of a Database Node
In order to initialy cluster component databases a high level description of the meta-data content of a database must rst be developed. To demonstrate this consider the previous example of the Universal Accreditation database, which deals with academic institutions and accreditation processes. This database contains entities such as courses, committees, (accreditation) processes, etc. We use a variant of an information retrieval (IR) technique called, star technique, where a term is selected and then all terms related to it are placed in a class[10]. Terms not yet in a class are selected as new seeds until all terms are assigned to a class. The variant of the star technique that we are using starts with a term represented as an abstract class (term descriptor class), then an additional term that is related to the term selected is represented as a another class and is connected to the selected term. The new term is then selected as a pivot and the process is repeated until no new terms can be added. In this way a context graph created for a speci c database schema. For example, the context graph for the Universal Accreditation component database (Figure 5) contains nodes
262
M.P. Papazoglou and S. Milliner
Fig. 4.
More contextual information regarding the term course.
which correspond to the abstract term descriptor classes committee, institutions, courses etc., while the context graph edges depict inter{connections (association, generalization, specialization or containment) between the terms within this particular database. Term interrelations are determined on the basis of a reference lexicographic substrate that underlies all the GCs in the network. For this purpose we use the lexicographic system4 WordNet [13] that supports semantic term matching through the use of an extensive network of word meanings of terms connected by a variety of textual and semantic relations. To facilitate clustering and discovery of information, we require that a component database (e.g., Universal Accreditation) can be totally described in terms of three sections which contain a synoptic description of the meta-data content of the database; associations between meta-data terms in the form of a semantic4
This lexicographic tool is presently used only for experimental purposes and will be replaced by an appropriate subject gateway in the near future.
Subject-Based Organization of the Information Space in Multi-database Networks
PUBLICATIONS EDUCATION & GC TRAINING PROVIDERS Accredi tation
5 Accredi tation DB
Applica tion
. Subject
List of keywords
GC
Institu tion
Course
10
. .
7 GOVERNMENT DEPTS GC
to other databases in same GC
WordNet Semantic Network Fig. 5.
Describing a component database.
net; and nally, links from these descriptions to other related databases in the network. This information can be viewed by users of the system once they have chosen a component database that potentially matches their interests (see section 5). Figure 5 illustrates that each database node contains the following sections: a feature descriptions, a context graph, and a GC connections section. The feature descriptions section contains information about terms, composition of terms, remarks about the meaning of terms, hypernym, hyponym, antonyms-of, part-of, member-of (and the inverses), pertains-to relations and lists of keywords. This section may also include certain details such as: geographical location, access authorization and usage roles, explanations regarding corporate term usage and de nitions, domains of applicability and so on. The feature descriptions entries are partially generated on the basis of WordNet and contain information in the form represented in Figures 2, 3 and 4. The context graph section contains a non{directed graph which connects term synopses (in the form of term descriptor classes) found in the Universal Accreditation database schema. Except for viewing purposes, the term descriptor nodes and their link structure are used
264
M.P. Papazoglou and S. Milliner
in the clustering of databases to form the generic concepts. Each of the term descriptor nodes de nes (in conjunction with its respective entry in the feature descriptions window) a common structured vocabulary of terms { describing the term in question, e.g., course, { and a speci cation of term relationships within that particular subject. Finally, the GC connection section shows how the Universal Accreditation database is related, i.e., content link weights, to other GCs in the network.
4.2 Similarity-based Clustering of Database Nodes Similarity-based clustering of database schemas organizes databases into related groups based on the terms (term descriptor nodes) they contain and the link structure of their context graphs. Our clustering algorithm determines the similarity between two graphs (representing two di erent database schema meta-data) based on both term similarity and link similarity factors. This is accomplished in two steps. Firstly, a pairwise-similarity of nodes in two context graphs is computed. From this an initial \pairing" of the nodes is determined. In the second step a comparison of the link structure of two context graphs is made based on the inter{node pairings and a semantic distance value is calculated. We chose this term/link similarity-based algorithm because it is relatively easy to implement and avoids generating very large clusters.
Term-based Similarity: this is calculated using cluster analysis techniques [5]
to identify co{occurrence probabilities { representing the degree of similarity { between two discrete terms. Our similarity metric is based on the meaning of the collection of terms representing the topical context (viz. semanticlevels) of a particular term, e.g., course, and the synonyms of these, see Figure 3. The comparison is based on: a conversion of each context graph node (e.g., term descriptor) Committee, Process, Subject, Course, etc. (see Figure 5) to a corresponding matrix of noun terms (containing the entire topical context of a term); and a subsequent comparison of terms within these matrixes. A matrix an;m of (noun) terms, representing the topical context of a particular term, ai;1 (course say), will correspond to the name of the term descriptor in the context graph. The synonyms of this term will be ai;2 , ai;3 ... ai;m (course-of-study, course-of-lectures). Terms ai,x;j (x > 0), e.g., education, educational-activity, will be more general than terms ai;j , while terms ai+x;j will be more speci c, e.g., CS-course. In the nal step, all synonyms for these terms are generated to produce the node's a complete topical description matrix an;m for a speci c term. Similarity analysis is mainly based on statistical co{occurrences of term descriptor objects based on techniques which has been successfully used for automatic thesaurus generation of textual databases [5], [21]. In fact we base our term-based similarity on the improved cosine formula [21] which is used to calculate the semantic distance between the vector for an item in a
Subject-Based Organization of the Information Space in Multi-database Networks
265
hierarchical thesaurus and the vector for a query item. To provide the right ontological context for semantic term matching, we use again the massive semantic net WordNet [13]. Comparison of the conceptual structure of two context graphs: to determine the structural and semantic similarity between two graphs, we based our algorithms regarding conceptual similarity between terms on heuristics{ guided spreading activation algorithms, and on work in the information retrieval area presented in [20]. These approaches take advantage of the semantics in a hierarchical thesaurus representing relationships between index terms. The algorithms calculate the conceptual closeness between two index terms, interpreting the conceptual distance between two terms as the topological distance of the two terms in the hierarchical thesaurus. During this process similarity between nodes (term descriptors) is established by considering the edges separating the nodes in the context graph as well as the actual graph structure. Some early results regarding the comparison and clustering process are described in [15].
Once similarity between nodes has been established context graphs are aggregated to create GCs. The aggregation of the context graphs from various component databases, results in the clustering of inter{related database schemas, see
266
M.P. Papazoglou and S. Milliner
Figure 6. The aggregation algorithm employed does not integrate the aggregates, as is the usual case with other approaches [8], but rather links descriptor classes at the GC level with corresponding term descriptor classes in its underlying cluster of database context graphs. Again this association is performed on the basis of the reference lexicographic substrate (WorNet). For each database cluster, a GC is created to represent the area of interest (or concept) that the group embodies, e.g., Education and Training Providers GC for the Employee Training, Accreditation, and Government Education Center databases as depicted in Figure 2.
5 Schema Term Navigation and Querying Information elicitation spans a spectrum of activities ranging from a search for a speci c data-item(s) (contained in possibly several component databases) to a non-speci c desire to understand what information is available in these databases and the nature of this information. 5.1
Navigation Techniques
There are two basic modes in which searching of the system may be organized. These search modes depend upon the nature of the information a user is attempting to access, and how this information relates to the database that user is operating from. Serendipity, exploration and contextualization are supported by means of indexing based upon terms contained in the component database context graphs. In such cases the user is interested in nding out about a particular topic rather than a speci c information (schema) item. We call this former form of exploration index-driven. Alternatively, if a user is seeking data which is closely related or allied to her/his local database, then searching may be organized around the weights of content links of this database to other GCs in the network. We refer to this form of exploration as concept-driven. Conceptdriven querying is the subject of a previous publication [18]. In this paper we will concentrate on index-driven exploration and on the querying of schema-related information. Index-driven navigation allows the users to deal with a controlled amount of material at a time, while providing more detail as the user looks more closely and is related to the dynamic indexing schemes and incremental discovery of information requirements for information elicitation. In order to traverse the index a user will have to decide on a number of key request terms, and then select synonyms or more general (and perhaps more speci c) derivatives of these key terms. The resulting query structure - generated on the basis of terms extracted from WordNet entries - can then be compared against the context graph structure of component databases. User speci ed term comparison starts at the top of the GC generated index and gradually percolates down to the required level of speci city by following the terms at each level. Figure 7 depicts this process in terms of a user query
Subject-Based Organization of the Information Space in Multi-database Networks
267
WordNet Generated Terms COURSE
Index on top level (most general) terms
(act, human, action, human activity)
User Query (activity, .....)
(education, educational activity)
(COURSE, course of study, course of lectures)
GC pointer to DB Context Graph
COURSE
Term 1 Term 2 Term . 1 Term . 2 . . .Term N . Term N
Database Context Graph
Database Database cluster cluster
Database cluster
Asociated Component DBs
Component Database Description
Fig. 7.
Accessing the index.
requesting information about courses at various institutions. Here we assume that the user has already speci ed that s/he is interested in the contents of the Education & Training GC. The graph of the user's query supplied terms contains a node Course and this term is used to traverse the GC generated index and arrive at the collection of databases which include this term (or its aliases) in their own descriptions. The index-driven navigation process starts with the most general terms possible, e.g., act, human activity, that correspond to the requested query term (course). These terms are generated by the GC (via the WordNet) and are presented to the user for selection. Once the user has selected a general term, most speci c terms are revealed, e.g., education. Once a GC term matching a user supplied term is selected, a link is established with the context graphs of all component databases containing the desired term (or its aliases). In this way the user can obtain contextual information and possibly a partial view of potentially matching databases and then s/he can decide whether a candidate database is useful or not. This hierarchical form of schema term navigation guarantees that a user supplied term correlates semantically with the content of the component databases underlying a GC cluster. The process is then repeated for all the other
268
M.P. Papazoglou and S. Milliner
terms in the user's query graph (i.e. the remaining unlabeled nodes in Figure 7). Thus, by matching the user query graph nodes to semantically equivalent GC terms, we can infer a number of component databases that are most closely associated to the user query.
5.2 Querying of Domain Meta-Data When the user needs to further explore the search target, intensional, or schema queries [17] { which return meta{data terms from selected schema terms { can be posed to further restrict the information space and clarify the meaning of the information items under exploration. Such domain-speci c queries should not be confused with queries which target the data content of the component databases (to which we refer to as distributed queries/transactions). Intensional queries are particularly useful for assisting users who are unfamiliar with the vocabulary of terms that can be used in connection with distributed queries/transactions or with the range of information that is available for responding to distributed queries. Sample intensional queries related to the GC in Figure 4 may include the following: query-1: Find the set of common super-terms of course. query-2: Find all terms more speci c than course and all their parts under sense education. query-3: Find the smallest common super-term of course of lectures and workshop. query-4: Find all parts of the term course. query-5: Which are the common properties of refresher course and seminar? query-6: Find all terms which contain the properties lesson and classroom project. query-7: What is the de nition of the term refresher course? All of the above queries - except for the last one - are rather intuitive. The last query returns a narrative description of the requested term in English (if available). Finally, when users feel suciently informed about the contents and structure of component database schema terms they have explored, they can pose meaningful distributed database requests which target the data content of the relevant component databases.
6 Experimentation The framework that we described in this paper is being implemented on Sun SparcStations under Solaris 2 using GNU C++ and CGI scripts. In order to evaluate automated clustering a test platform based on the clustering of about 100 networked databases has been created. There are two basic areas of experimentation being pursued. Firstly, there is the question of how well the initial automatic clustering of databases based on each component databases description
Subject-Based Organization of the Information Space in Multi-database Networks
269
can be performed. That is, the scalability question of nding appropriate initial relationships in the presence of large numbers of information sources. The types of experiments performed here are somewhat allied with the eld of information retrieval and clustering. The second set of experiments, on the other hand, deals with the processing and communications necessary to support the underlying distributed structure by which the generic concepts and their inter-relationships are implemented, queried and updated. This second group of experiments thus has its roots in the elds of distributed/parallel processing and communications performance. In a similar vein to IR experiments, the rst set of experiments are based on the notion of retrieval and accuracy (as de ned within IR). To achieve this, a collection of a hundred relational databases has been procured from a large organization's collection of information systems. A manual clustering of these was then performed by a domain \expert" who had full intimate knowledge of the organization's environment. This clustering was essentially based on where each database tted into the various departments within the organization, and how these departments interacted/overlapped { the latter being identi ed via analysis of database table usage within the various departments. Thus, we clustered databases based on the actual usage of data from the various information components as dictated by the organization of the environment that the databases were set up to model in the rst place { but in a macro (organization wide) sense rather than a micro (department based) sense. Experiments have been performed (and continue to be performed) to: 1. identify if automatic clustering can achieve a \near perfect" initial organization of the database collection - or at least be statistically signi cantly better than \raw" automatic clustering, which involves the identi cation of an appropriate heuristic for measuring the similarity between database descriptions; 2. compare results against other standard automatic clustering packages (e.g., those found in IR); 3. determine what set of descriptive \primitives" are essential (and minimal) to achieve a satisfactory degree of clustering; 4. determine the \robustness" of the description process { i.e., give some indication of how much variation there can be within a description before the automatic clustering becomes unsatisfactory. This last experiment is important as it must be remembered that di erent people may be responsible for the construction of di erent database descriptions. Thus, the automatic clustering must be relatively robust in terms of the way di erent people may describe the same object. It is expected that, given all descriptions will be generated using the same thesaurus, the system should prove relatively good at detecting di ering descriptions of a single object. Currently, experiments have been performed using a \full" database description involving the synonyms, generalizations and terms senses, as well as the structural relationships between these terms, see Figure 4. Initialy, the term
270
M.P. Papazoglou and S. Milliner
matching component was based on the standard similarity metric proposed by Dice [5], and the structural similarity was based on the notion of spreading activation energy [15]. It was found, however, that the accuracy and retrieval of this particular approach was not signi cantly better than the clustering of the \raw" database descriptions using Dice's method directly. Upon analysis it was discovered that performance was degraded due to the un-directed nature of the context graph. Thus, in a subsequent set of preliminary experiments, the notion of spreading activation energy was dropped, and a ranking of similarity based on the hierarchy of the graph was introduced. This resulted in a huge improvement in the retrieval and similarity gures which indicated the automatic clustering to be signi cantly better than the base-line clustering.
7 Summary and Future Work This paper described the fundamental aspects of a scalable, semantically oriented, con gurable distributed information infrastructure that supports information discovery and retrieval across subject domains in networked databases. The proposed logical architecture extracts semantics from database schemas and creates dynamic clusters of databases centered around common topics interest (viz the generic concepts). Large-scale searching is guided by a combination of lexical, structural and semantic aspects of schema terms in order to reveal more meaning both about the contents of a requested information item and about its placement within a given database context. To surmount semantic-drifts, the terminology problem and enhance database retrieval, alternative search terms and term senses are suggested to users. This architecture enables users to gather and rearrange information from multiple networked databases in an intuitive and easily understandable manner. Experience with this con guration suggests the clustering mechanisms used provide a valuable discovery service to end users, and that the logical organization used supports the ability of the system to scale with modest increases in GC label sizes. Future work addresses the semi-automatic generation of link weights based on term co-occurrences using statistical/probabilistic algorithms. In IR these algorithms use word and/or phrase frequency to match queries with terms [5]. In the current prototype link weights are established at a clustering phase on a tentative basis only. However, it is expected that during execution link weights to GCs may need to be updated (strengthened or weakened) over time depending on interaction, new GCs may be formed, and existing GCs may need to merge. The next suite of experiments to be performed will deal with the characteristics of the link weight update and GC split/merge processes. From this policies will be developed (e.g. delayed/batch updating of GC information), and then evaluated.
References 1. Arens Y., et al. \Retrieving and Integrating Data from Multiple Information Sources", Int'l Journal of Cooperative Information Systems, 2, 2, (1993).
Subject-Based Organization of the Information Space in Multi-database Networks
271
2. Bowman. C. M., et al. \Harvest: A Scalable, Customizable Discovery and Access System", Univ. of Colorado - Boulder, CS Dept., techn. report CU-CS 732-94, (1995). 3. Bright M., Hurson A., Pakzad S. \Automated Resolution of Semantic Heterogeneity in Multidatabases" ACM ToDS, 19, 2, (1994). 4. Castano S., De Antonellis V. \Semantic Dictionary Design for Database Interoperability", 13th Int'l Conf. on Data Engineering, Birmingham, April (1997), 43{54. 5. Everitt B. \Cluster Analysis", Heinemann Educational Books Ltd., Great Britain, (1981). 6. Kahle B., Medlar A. \An Information System for Corporate Users: Wide Area Information Servers", The Interoperability Report, 5, 111, (1991). 7. Kahng J., McLeod D. \Dynamic Classi cational Ontologies: Mediation of Information Sharing in Cooperative Federated Database Systems", in Cooperative Information Systems: Trends and Directions, Papazoglou M. P., Schlageter G. (eds), Academic-Press (1997) 179{203. 8. Kashyap V., Sheth A. \Semantic Heterogeneity in Global Information Systems: the Role of Metadata, Context and Ontologies", in Cooperative Information Systems: Trends and Directions, Papazoglou M. P., Schlageter G. (eds), Academic-Press (1997) 139{178. 9. Kirriemuir J. et al., \Cross-Searching Subject Gateways", D-Lib Magazine, January (1998). 10. Kowalski G. \Information Retrieval Systems: Theory and Implementation", Kluwer Academic Publishers, (1997). 11. Mauldin L.M., Levitt J.R. \Web-agent related Research at the CMT", Procs. ACM Special Interest Group on Networked Information Discovery and retrieval: SIGIR'94, August (1994). 12. McLeod D., Si A. \The Design and Experimental Evaluation of an Information Discovery Mechanism for Networks of Autonomous Database Systems", 11th Int'l Conf. on Data Engineering, Taiwan, Feb. (1995) 15{24. 13. Miller G. \WordNet: A Lexical Database for English", Communications of ACM, 38, 11, Nov. (1995). 14. Milliner S., Bouguettaya A., Papazoglou M.P. \A Scalable Architecture for Autonomous Heterogeneous Database Interactions", 21 Int'l Conference on Very Large Databases, Zurich, Switzerland, Sept. (1995). 15. Milliner S., Papazoglou M., Weigand H. \Linguistic Tool based Information Elicitation in Large Heterogeneous Database Networks", NLDB '96 Natural Language and Databases Workshop, Amsterdam, June (1996). 16. \OMNI, Organizing Medical Networked Information", http://omni.ac.uk/ 17. Papazoglou M.P. \Unraveling the Semantics of Conceptual Schemas", Communications of ACM, 38, 9, Sept. (1995). 18. Papazoglou M.P., Milliner S. \Pro-active Information Elicitation in Wide-area Information Networks", Procs. of the Int'l Symposium on Cooperative Database Systems for Advanced Applications, World Scienti c, Japan, Dec. (1996). 19. Pinkerton B. \Finding what People Want: Experiences with the WebCrawler", Procs. 1st Int'l Conference on the WWW, Geneva, May (1994). 20. Rada R., Bicknell E. \Ranking Documents Based on a Thesaurus", Journal of the American Society for Information Science, 40, 5, May (1989). 21. Salton G.E, Buckley C. \Term-Weighting Approaches in Automatic Text Retrieval", Information Retrieval and Management, 24, 5, (1988), 513{523.
272
M.P. Papazoglou and S. Milliner
22. Schatz R.B., et. al \Interactive Term Suggestion for Users of Digital Libraries", 1st ACM International Conf. on Digital Libraries, Bethesda MD, March (1996), 126{133. 23. Sheldon M.A. \Content Routing: A Scalable Architecture for Network-Based Information Discovery", PhD thesis, MIT, Dec. (1995). 24. Sheth A., Larson P. "Federated Database Systems for Managing Distributed, Heterogeneous and Autonomous Databases". Computing Surveys, 22, 3, Sept (1990). 25. \SOSIG: The Social Science Information Gateway", http://www.sosig.ac.uk/ 26. Wiess R., et al. \HyPersuit: A Hierarchical Network search Engine that Exploits Content-link Hypertext Clustering", 7th ACM Conf. on Hypertext, Washington DC., March (1996).
MUSE - An Interactive Networked Multimedia Applications Specification Environment with E-LOTOS Translator Luciano Paschoal Gaspary Maria Janilce B. Almeida Universidade Federal do Rio Grande do Sul Instituto de Informática Curso de Pós-Graduação em Ciência da Computação Campus do Vale, Bloco IV – Bento Gonçalves, 9500 – Agronomia – 91591-970 Porto Alegre, RS – Brazil E-mail: {paschoal, janilce}@inf.ufrgs.br Abstract. This work presents MUSE, a graphical environment for modeling interactive networked multimedia applications. Through an advanced graphic interface and a new highlevel authoring model, it is possible to create complex systems in a fast and intuitive way. The authoring model proposed in this work and adopted by the environment deals with media objects distributed in a computer network, allowing the definition of acceptable presentation delay thresholds and alternative media objects. Due to the large expressiveness of the model, however, specifications with logical and temporal inconsistencies may be generated. For this reason, the tool also provides E-LOTOS specifications, which may be used to analyze and verify the temporal requirements defined by the author.
1
Introduction
The 90’s have been known by the use of multimedia applications in several fields of the human activity such as education, medicine and entertainment. These applications have become increasingly sophisticated along the time, and nowadays they are executed in distributed environments, operating transparently in heterogeneous platforms. The possibility of having an application with its media objects dispersed in a network influences the creation and modeling of such applications. Users must provide the authoring tools with information like temporal restrictions, defining acceptable delay thresholds to the presentation of the elements that compose the system and establishing the presentation of alternative media objects. The definition of these restrictions is accomplished based on a synchronization model, which dictates the rules about how the media objects of an application can be related in time. Several synchronization models have been proposed [1]. Most of them are both flexible and very expressive. That is the reason why the resulting specifications can be source of incoherences, where the logical and temporal consistency of the involved media objects can not be assured. An alternative would be to use directly a formal description technique (FDT) to describe the applications, making its analysis possible and so guaranteeing its consistency. The disadvantage of this direct usage, however, is the high complexity inherent to FDTs. So, the need of having a structured high-level model to specify interactive networked multimedia B. Pernici and C. Thanos (Eds.): CAiSE’98, LNCS 1413, pp. 273-287, 1998. Springer-Verlag Berlin Heidelberg 1998
274
L.P. Gaspary and M.J.B. Almeida
applications becomes evident. The resulting specifications shall then be translated to an FDT, so that verification and simulation methods can be applied to them. In this context, an interactive networked multimedia applications authoring model was created. MUSE (MUltimedia Applications Specification Environment) was developed to support this model, allowing the user to easily define a multimedia presentation according to the MHEG-5 standard [2]. The adoption of MHEG-5 allows multimedia information to be shared without worrying about the platform or operating system used, providing specification and development of portable applications. To make the validation process of the specifications possible, the environment automatically generates E-LOTOS specifications. This work is part of DAMD (Distributed Multimedia Applications Design) project, sponsored by the Brazilian research council. Its main objectives are to provide a methodology to completely cover the distributed multimedia applications development cycle and to allow authors who are not expert in formal methods to easily develop their applications. The project was developed according to figure 1. MUSE, in a certain way, centralizes the process that comprehends modeling and presentation of applications. Specifications created by the user are validated and the obtained results are presented to him in a quite readable way in the own tool. The specification-validation process repeats until the incoherences are eliminated. After that, MHEG-5 applications are generated and can be executed by the engine. PRGHOLQJ SURFHVV SUHVHQWDWLRQ XVHU 086(
(/2726 VSHFLILFDWLRQV
0+(* (QJLQH
UHVXOWV
6LPXODWLRQ9HULILFDWLRQ
Fig. 1. Structure of the DAMD project
This paper is organized as follows: section 2 presents important aspects to be considered in the applications authoring process, relating them to some multimedia synchronization models pointed by the literature. This section also presents the proposed authoring model. In section 3 basic aspects of the E-LOTOS FDT are presented, as well as a mechanism to represent specifications generated by the authoring model in this formal technique. Section 4 illustrates the functionality of the environment and in section 5, one can read the final considerations.
2
Proposed Authoring Model
The specification of multimedia applications is accomplished with base on three fundamental aspects: logical structuring, establishment of temporal relationships and spatial definition among the elements belonging to the application. The logical structuring is concerned to offer abstraction mechanisms, providing a wide and structural view of the application. The specification of the temporal behavior involves
MUSE - A Multimedia Applications Specification Environment
275
the definition of synchronization relations among media objects. The spatial synchronization cares about adjusting the positioning of the visible media objects according to the output devices (video). The temporal relations are established according to a synchronization model, which imposes rules on how these elements can relate to each other. Several models have been proposed in the literature. One of the most adopted by existent authoring tools is the time-line based one [3]. However, it presents many limitations such as the difficulty both to modularize the application and to establish relations among elements with variable or unknown duration like user interaction [4]. The hierarchical model also presents deficiencies. The most important one is that the construction and reading process of the specifications is not natural. It is not clear the order in which the media objects will be presented. Besides, the model does not allow the establishment of some synchronization rules [1], which restricts the expression power of this model. Models based on references to points are not adequate to model distributed multimedia applications, because there is no an explicit time notion. Thus, temporal restrictions can not be expressed and temporal irregularities (common in distributed systems) are ignored in this model. In synchronization models based on Petri nets, it is possible to specify most of the important synchronization rules required for modeling multimedia applications [5]. Among the models up to now presented, this one provides the largest expression power and flexibility. Moreover, as Petri net is a formal model, it makes applications analysis possible, allowing its consistency to be guaranteed. Its largest disadvantage, however, is its complexity; the manipulation of large specifications may become difficult because of the state explosion problem. In this work, an authoring model that joins mechanisms for logical structuring the applications to a synchronization model similar to HTSPN is proposed. The logical structuring level is based on the concept of scenes and groups, providing a broad view of the application. The definition of temporal synchronizations is done in each scene by means of a simplified graph. The spatial synchronization allows media objects to be positioned considering the output device (see figure 2). 2.1 Logical Structuring The complexity of multimedia applications increase according to the growth of the number of involved media objects and, consequently, to the several temporal relationships established among them. This is the fundamental reason why the specification of these applications in only one plane is inappropriate. To solve this problem, the concept of scenes was incorporated into the model considering the MHEG-5 standard. Multimedia applications can be organized as a group of scenes related by events, which provide the navigation among them. Each of these scenes can be seen as a black box with an internal behavior that, under certain conditions, enables the presentation of other scenes. The use of this concept, however, does not solve completely the problem of complexity, since a specification with many scenes will be hardly understood. Trying to make easier the understanding of so large applications, a hierarchy mechanism was added to the model through the concept of group of scenes. The top of figure 2 illustrates the logical structure of an application, composed by four scenes (Scene1,
276
L.P. Gaspary and M.J.B. Almeida
Scene2, Scene3 and Scene4). Three of them (Scene2, Scene3 and Scene4), due to the cohesion established among them, were gathered in Group1. The arcs that link groups and scenes in the logical structure do not represent temporal synchronizations, but choices. For example, a scene A tied up to two scenes B and C indicate that the temporal behavior of the scene provides two ways for the application to evolve: either to B or to C, only depending on the dynamic behavior of the application. This evolution is materialized by the use of the transition icon, to be mentioned in the following section. $SSOLFDWLRQ 6FHQH
*URXS
(QG
/RJLFDO 6WUXFWXULQJ 6FHQH
6FHQH 6FHQH
3UHVHQWDWLRQ LFRQ
,QWUR
$UF
7UDQVLWLRQ WR *URXS
0DFKLQHV 6SDWLDO YLHZ GXULQJ WKH SUHVHQWDWLRQ RI 0DFKLQHV
6SDWLDO DQG 7HPSRUDO 6\QFKURQL]DWLRQ
«
%XWWRQ
7UDQVLWLRQ WR (QG
7UDQVLWLRQ WR 6FHQH
9LGHR 6SDWLDO YLHZ GXULQJ WKH SUHVHQWDWLRQ RI 9LGHR
2XWSXW GHYLFH
Fig. 2. Structure of an interactive multimedia application
Usually, there are media objects whose presentation embraces several scenes. With the purpose of increasing the expressiveness of the model, the possibility of representing media objects shared by several scenes was created. Figure 3 shows an application organized in three scenes (Scene1, Scene5 and Scene6) and a group (Group1). The image Logo is presented simultaneously to the whole application, and Instructions, during the presentation of Group1 and Scene5.
MUSE - A Multimedia Applications Specification Environment
277
,QVWUXFWLRQV 6FHQH
*URXS
6FHQH
6FHQH
/RJR
Fig. 3. Media objects shared among scenes and groups
From the authoring process perspective, the proposed structure facilitates the reuse of scenes and groups that repeat in different specifications. Besides, the model allows the development of templates – basic pre-constructed scenes – whose utilization makes the specification process evolving and incremental. One can have a set of templates, so that the specification process, in this case, is reduced to joining these different scenes, lessening drastically the development efforts. 2.2 Temporal Synchronization The temporal synchronization of an application, as mentioned previously, refers to the ordering of the presentation of its media objects in time. Each media object has a presentation duration that may or may not be foreseen, depending on its nature. The following topics present how the synchronization relationships can be established. Basic Synchronization. Media objects can be presented sequentially or simultaneously. In the sequential presentation, the playout of a media object depends on the end of another's. In figure 2 both types of basic synchronization appear. In Scene1, the presentation of a text (Intro) is followed by the presentation of an image (Machines). In Scene4, there is the simultaneous presentation of a video (Video1) and a button (Button). Event Duration and Delay. A minimum and a maximum duration of presentation are associated to each media object. In the case of an image or a text, these values are equivalent because they are time-independent media objects. When one deal with media objects like audio and video, however, it is important to determine both a minimum and a maximum presentation duration, since these media objects will be hardly presented at the nominal rate due to problems like network traffic. The representation of these durations is given by an interval. To make the modeling of a delay between the presentation of two consecutive media objects possible, a special icon can be used. It does not have any media object associated to itself but only a specific value representing how long it has to wait to start the presentation of its successive media. Figure 4 illustrates three slides (Slide1, Slide2 and Slide3) being presented sequentially with a delay of three seconds between the first and the second and a delay of five time units between the second and the third one.
278
L.P. Gaspary and M.J.B. Almeida
6OLGH
6OLGH
6OLGH
Fig. 4. The delay icon
User Interaction and Scene Transition. User interaction corresponds, for instance, to a button click or an object selection. It is represented in this model as a constructor whose presentation duration is uncertain, varying between the minimum and maximum values associated to it. When the maximum threshold is reached, the scene continues with its presentation. It is still possible to specify a button without maximum duration; in this case, its evolution will only happen after the interaction. The user interference is normally associated to a scene transition. Transition is the constructor that makes the navigation among scenes possible. Its execution involves both the immediate suspension of the presentation of all the media objects belonging to the current scene and the beginning of a new scene presentation. In Scene4 (see figure 2), the transition to Scene3 occurs after the hitting of the button; if the video (Video1) is still being presented at that instant, it is interrupted. The connections described in the logical structure and the transitions used in the temporal structure must be consistent to each other. In Scene1, for example, the only acceptable transition is to Group1, once in the logical structure the scene only has connection to the icon that indicates this group. Synchronization Points. Synchronization points allow the beginning of the presentation of one or more media objects to be associated to different policies related to the end of the presentation of other media objects that converge to these points. To simplify the graphical representation of the authoring model, synchronization points involving only two presentations are not shown. For instance, in figure 4 the synchronization points between Slide1 and the delay and between the delay and Slide2 are not presented. $XGLR
$XGLR $XGLR
>@
$XGLR >@
9LGHR
9LGHR
PDVWHU
>@ >@
D 6\QFKURQL]DWLRQ SRLQW
>@ E 0DVWHU ILULQJ UXOH
Fig. 5. Synchronization point and firing rules
>@
MUSE - A Multimedia Applications Specification Environment
279
To increase the specification power, the model has adopted some policies widely commented in the literature. They allow the association of different behaviors to the synchronization points [6]. For simplification, the model only supports three of them: −
Master: the synchronization point is fired when the presentation of a master media object is finished, interrupting all the others. This rule could be used in the example of figure 5a above if one wishes that the end of the presentation of Video (master) causes the interruption of Audio1, starting Audio2 (see figure 5b). The master media object is identified by the presence of the character m or the word master close to it.
−
Earliest: the synchronization point is fired when the presentation of the first media object is finished, resulting in the interruption of the others. This rule is graphically represented by the presence of the character e or the word earliest close to the synchronization point.
−
Latest: the absence of an indication close to the media object or to the synchronization point means that all the media objects that precede this point will be executed (or they will conclude due to the elapsing of their maximum presentation duration) before the synchronization point is fired (figure 5a).
Instants of Synchronization. In MUSE, the synchronization among media objects in other instants than the beginning and end of its presentations requires the division of these media objects in parts, creating a set of segments. The granularity of this division is associated to the precision degree desired for the synchronization. Figure 6 shows the synchronization of two subtitles (Subtitle1 and Subtitle2) with a video (VD), where the latter is divided into four segments. The first subtitle is presented simultaneously to the second video segment and the second subtitle together with the third segment. 9'> @
9'>@
9'> @
>@
>@
>@
6XEWLWOH
9'> @
>@
6XEWLWOH
Fig. 6. Synchronization of a video with subtitles
2.3 Spatial Synchronization The spatial synchronization allows the author to visualize the positioning of the visible media objects of a scene. It is not possible to accomplish the spatial structuring considering a certain time elapsed after the beginning of the scene execution. It is so because each of the executions of the application, due to the acceptable temporal variations, the media objects can be presented in different instants. For this reason, the spatial synchronization is always accomplished with relation to the presentation of a media object. The spatial arrangement of the media objects of Scene1 (see figure2) during the presentation of Machines, for example, will only allow the bitmap
280
L.P. Gaspary and M.J.B. Almeida
Machines to be organized. On the other hand, the spatial view of Scene4 during the presentation of Video1 will present the media objects Video1 and Button. The appearance of Button occurs because it is defined to be simultaneously presented with Video1. 2.4 Example of Model Usage The example illustrated in figure 7 models the application proposed in [1], where initially a video (VD) and an audio (AU) are executed simultaneously. Following, a recorded user interaction (RI), a sequence of three slides (P1-P3) and an animation (ANI) which is partially commented by an audio sequence (Audio2) are presented sequentially. During the animation, a multiple-choice question is presented to the user (Interaction). If the user makes the selection, a final image (P4) is presented. This is just one of several ways of representing this application. The ease in understanding it is obtained mainly by the user's good sense in the moment of its specification. ,QLWLDO*URXS
6FHQH
6FHQH
6FHQH
9'B3
9'B3
P
>@
>@
6FHQH
6FHQH
9'B3
P
P
>@
$8B3
>@
(QG
6FHQH
>@
$8B3
>@
5,
3
3
3
>@
6FHQH
$8B3 6FHQH
$XGLR $1,B3
>@
,QWHUDFWLRQ
>@
>@
3
>@
$1,B3 (QG
Fig. 7. A simple example of the model usage
MUSE - A Multimedia Applications Specification Environment
3
281
Representation of Multimedia Applications in E-LOTOS
The formalization of specifications is important for the process of their validation. The proposed authoring model, due to its high flexibility and expressiveness, allows both temporally and logically incoherent specifications to be defined. The analysis process detects, for example, conflicts in resources usage and tests if the application’s end can be reached from all the possible navigation paths. Thus, specifications described by an author according to the model presented in the previous section are translated to a formal representation, analyzed and the obtained results are presented to the user, who will make the necessary adjustments. The formal description technique E-LOTOS (Enhancements to LOTOS) [7] is an enhanced version of LOTOS and is in standardization process. The main innovation of the language is the incorporation of quantitative time notion, allowing the definition of instants in which actions or events may happen. This is a fundamental feature for representing multimedia applications and, for this reason, E-LOTOS was chosen to formally represent them. The representation of multimedia applications is hierarchical and considers the four essential elements of the authoring model: application, group, scene and media object. All these elements are modeled as processes that evolve according to previously established synchronization relationships. The way of formally represent multimedia applications commented in this section is based on the approach presented in [8]. Further details are presented in the following topics. 3.1 Data Representation and Root Process Instantiation Data representation is done by means of a library called classes, which define data types for all possible media objects. There are types like BitmapClass, StreamClass and SwitchButtonClass, whose definition is based on their respective MHEG-5 classes. For example, the fields of BitmapClass are the media object, its position in the output device and its dimensions. The application is started from the instantiation of the root group process. After that, the application is indeed able to evolve. 3.2 Group Representation In the representation of groups, the hiding operator is used. Taking the example of figure 8, one can see that some internal events like the beginning of both Scene2 (s_Scene2) and Scene3 (s_Scene3) are not visible outside the process (1). These events are used to synchronize the presentation of the scenes belonging to InitialGroup. The synchronization is modeled with the par operator (2). For instance, the beginning of Scene2 is associated with the end of Scene1 (s_Scene2) (3 and 4). The same occurs with Scene2 and Scene3: the beginning of the latter is synchronized with the end of Scene2 (s_Scene3) (4 and 5). The disabling operator must also be mentioned (6). As one can observe, the req_End event reaches all the processes of the group; it is used to model the end of the application. When it is generated (by a transition to end), groups and scenes are successfully terminated (6).
3.3 Scene Representation Scene modeling differs in many aspects from group representation. One of the differences is that scene processes instantiate media objects and restrictions instead of groups and scenes. The presence of the loop operator in the representation is another important difference (1) (see figure 9). It is used to allow a scene to be presented more than once, which may happen when the application is logically organized as a net. Figure 9 shows Scene2, previously instantiated in figure 8. The req_Res event is responsible for restarting the media objects of the current scene when a transition to another scene occurs. The code that models a scene transition is composed of three events: s_Trans, req_Res and f_Scene (see figure 10a). The former denotes the occurrence of the transition. The second invokes the media objects of the scene to be reset. The third one indicates the end of the scene presentation. As the transition is an endless process, it is also disabled by the occurrence of the req_End event. When the transition is to the end of the application, the req_Res event is replaced by the req_End event (see figure 10b).
MUSE - A Multimedia Applications Specification Environment
HQGORRS >! UHTB(QG H[LW HQGSURF D 6FHQH WUDQVLWLRQ
E 7UDQVLWLRQ WR WKH HQG RI WKH DSSOLFDWLRQ
Fig. 10. Representation of transitions
3.4 Basic Objects and Temporal Restrictions Basic or monolithic objects were defined by [3] and model the presentation of simple media objects. These media objects are defined by the occurrence of synchronous (beginning, end) and asynchronous (user interaction) events. Several combinations of these events can be formulated, but only eight are pointed as important in the definition of interactive multimedia scenes. This work presents three of these combinations (see table 1). The fourth object presented in this table (pSsSe Synchronous start Synchronous end) does not appear in [3]. It allows time-dependent media with both minimum and maximum presentation durations to be modeled. In the definition of the processes, the Data event was used to represent the presentation of the media object.
Figure 11 shows the representation of P2, which appeared in the definition of Scene2 in figure 9. The event req_End (3) can again be observed, because media objects are also always being executed (1); if there is a loop in the scene definition, some media objects may be executed more than once during the presentation of the scene. In the same figure, one can also see the effect of the occurrence of the req_Res event: the restart of the media object to its initial state (2). SURFHVV 3 >VB3HB3 'DWDUHTB(QGUHTB5HV@ H[LW LV 3 %LWPDS&ODVVG3 7LPH ORRS IRUHYHU LQ
S6V6H>VB3 HB3 'DWD@ 3G3 >!UHTB5HV
HQGORRS >! UHTB(QG H[LW
HQGSURF
Fig. 11. Representation of the media object P2
The authoring model and consequently the tool, to be described in the next section, provide the definition of three distinct temporal restrictions: WaitMaster, WaitLatest and WaitEarliest. Their E-LOTOS representation controls the end of the media objects presentation that converge to the synchronization point. Restrictions are not implemented in libraries because their behaviour depends on the number of media objects that converges to the synchronization point. Figure 12 shows the representation of the WaitEarliest restriction. SURFHVV :DLW(DUOLHVW>HB$HB%HB& HB5HVWULFWLRQUHTB(QGUHTB5HV@ H[LW LV ORRS IRUHYHU LQ HB$>@HB%>@HB& HB5HVWULFWLRQ#W>W @ >!UHTB5HV HQGORRS >!UHTB(QG H[LW HQGSURF
Fig. 12. Representation of the restriction WaitEarliest
MUSE - A Multimedia Applications Specification Environment
4
285
The Authoring Environment
The creation environment is divided in two units: media repository and specification area. At any moment, the user can insert media objects into the repository. This is done by browsing a local media object or referencing a remote one. Figure 13 shows two windows that allow, respectively, the incorporation of new media objects (New Media) to the application and the manipulation (Medias Palette) of already existent ones.
Fig. 13. Management of the media objects of the application
The specification area is composed of the scenes and groups of an application. Each scene is represented by both the temporal and the spatial views. The former allows the user to insert icons and synchronization points and to establish their relationship using arcs. The visible elements, used in the temporal synchronization, can be adjusted in the spatial view. Figure 14 shows the basic functionality of the authoring environment. The toolbar has shortcuts to its main functions (1). Two support windows can be observed: Specification Hierarchy (2) and Icons Palette (3). The former provides the user a general view of the application, presenting all the scenes and groups in a tree and providing a structured view of the relationships among them. In this case, the modeled application was the example presented in section 2 and is composed, therefore, of three scenes: Scene1, Scene2 and Scene3 (4). The latter, in its turn, provides mechanisms to visualize and edit the icons properties. In the same figure, the bitmap icon (P1) of Scene2 is selected and its specific properties are presented in the mentioned window. Icons that have an associated media object (audio, text, image, and video) present a special property called media. This property must be filled with a media object existing in the repository. In this example, icon P1 is associated to the media object Rio Bonito.
286
L.P. Gaspary and M.J.B. Almeida
1 2 4
3 5
6
Fig. 14. Interface of the authoring environment
In figure 14, one can also observe the specification of Scene2 (5). It is composed of video (RI) followed by the sequential presentation of three images (P1, P2 and P3). By the end of the presentation of the last media object, a transition to Scene3 occurs. These information are presented by the temporal view. At the same time, the spatial view of Scene2 taking the icon P1 as reference is showed (6). It is possible to move or resize the visible media objects. Their properties related to coordinates and dimensions are automatically updated. Time-dependent media objects, like video, can be divided in smaller segments, allowing the synchronization of other elements with specific points of them. The environment provides mechanisms that make the process of fragmentation of these media objects easy. MUSE also provides means to reuse scenes and groups. It can be done by browsing the group or scene to be retrieved. It is necessary to redefine the transitions, defining where the application should evolve to after its presentation. Finally, it is also important to highlight the functionality of E-LOTOS code generation. This is obtained through the special saving option Save as E-LOTOS.
5
Conclusions and Future Work
This work initially proposed a new model for specifying interactive networked multimedia applications. Besides, mechanisms for mapping this model to the ELOTOS language were presented. Finally, the developed environment was described. The main contribution of this work is, therefore, the construction of an environment turned to both ease of use and good expressiveness. At the same time, means to provide the formal representation of applications aiming at its analysis is also a great contribution.
MUSE - A Multimedia Applications Specification Environment
287
The model proposed distinguishes intentionally the concepts of logical structuring and temporal synchronization. The logical structure of the applications facilitates its organization in chapters, sections or in any other unit. For this reason, the application becomes modular, which contributes to lessen the complexity of the scenes and to avoid the occurrence of the state explosion problem. Future works include the creation of mechanisms that allow the user to define in the own environment parameters of quality of service, which will be used during the execution of the application. The possibility to define alternative media objects is also an important future task. It is important to highlight that the use of this environment integrated to the other tools under development in the project provides a complete framework, covering all the steps involved in the design of distributed multimedia applications: specification, verification and presentation. The ease of the authoring model and the use of a formal description technique to validate the applications turn the environment attractive and easy to use, without restricting the expressiveness of the environment.
References 1. G. Blakowski and R. Steinmetz. A Media Synchronization Survey: Reference 2. 3. 4.
5. 6. 7. 8.
Model, Specification, and Case Studies. IEEE Journal on Selected Areas in Communications, 14(1): 5-35, January 1996. ISO/IEC DIS 13522-5. Information Technology - Coding of Multimedia and Hypermedia Information, Part 5: Support for Base-Level Interactive Applications, 1995. N. Hirzalla, B. Falchuk and A. Karmouch. A Temporal Model for Interactive Multimedia Scenarios. IEEE Multimedia, 24-31, fall 1995. L. Soares e R. Rodrigues. Autoria e Formatação Estruturada de Documentos Hipermídia com Restrições Temporais. In Proc. of the 3rd Workshop on Multimedia and Hypermedia Systems, São Carlos, Brazil, May 1997. (In Portuguese) P. Sénac, R. Willrich, P. de Saqui-Sannes. Hierarchical Time Stream Petri Nets: A Model for Hypermedia Systems. In Application and Theory of Petri Nets, 1995. P. Sénac, M. Diaz and P. de Saqui-Sannes. Toward a formal specification of multimedia synchronization scenarios. Ann. Telécommun. No. 49, pp 297-314. ISO/IEC JTC1/SC21/WG7. Enhancements to LOTOS. Revised Working Drafts on Enhancements to LOTOS (V4), Project WI 1.21.20.2.3, January 1997. J. P. Courtiat and R.C. de Oliveira. Proving Temporal Consistency in a New Multimedia Synchronization Model. ACM Multimedia, Boston, 1996.
Information Extraction & Database techniques: a user-oriented approach to querying the Web Zoe Lacroix?1 and Arnaud Sahuguet?? 2 and Raman Chandrasekar? ? ? 3 IRCS, University of Pennsylvania CIS, University of Pennsylvania IRCS & CASI, University of Pennsylvania 1
2
3
Abstract. We propose a novel approach to querying the Web with a sys-
tem named AKIRA (Agentive Knowledge-based Information Retrieval Architecture) which combines advanced technologies from Information Retrieval and Extraction together with Database techniques. The former enable the system to access the explicit as well as the implicit structure of Web documents and organize them into a hierarchy of concepts and meta-concepts; the latter provide tools for data-manipulation. We propose a user-oriented approach: given the user's query, AKIRA extracts a target structure (structure expressed in the query) and uses standard retrieval techniques to access potentially relevant documents. The content of these documents is processed using extraction techniques (along with a exible agentive structure) to lter for relevance and to extract from them implicit or explicit structure matching the target structure. The information garnered is used to populate a smart-cache (an object-oriented database) whose schema is inferred from the target structure. This smart-cache, whose schema is thus de ned a posteriori, is populated and queried with an expression of PIQL, our query language. AKIRA integrates these complementary techniques to provide maximum
exibility to the user and o er transparent access to the content of Web documents. Keywords: Web, data model, query language, information retrieval & extraction, agents, cache, view.
1
Introduction
1.1 A user-oriented approach
The Web represents an immense reservoir of information, of great value if only we can manage it properly. This issue is a database concern since it involves
Institute for Research in Cognitive Science, University of Pennsylvania, Suite 400A, 3401 Walnut Street, Philadelphia PA 19104, USA { Work supported by NSF STC grant SBR-8920230 and ARO grant DAAH04-95-I-0169. ?? Department of Computer and Information Science, University of Pennsylvania, 200 South 33rd Street Philadelphia PA 19104, USA { Work supported by ARO grant DAAH04-95-I-0169 and ARPA grant N00014-94-1-1086. ? ? ? Institute for Research in Cognitive Science & Center for the Advanced Study of India, Suite 400A, 3401 Walnut Street, Philadelphia PA 19104, USA. ?
B. Pernici and C. Thanos (Eds.): CAiSE’98, LNCS 1413, pp. 289-304, 1998. Springer-Verlag Berlin Heidelberg 1998
290
Z. Lacroix, A. Sahuguet, and R. Chandrasekar
representing and accessing information. The standard database approach focuses on the source(s) and provides a query language based on a given organization (schema) of the data: this is the source-driven approach. There have been various proposals to apply the source-driven approach to querying the Web. Some are oriented more towards the explicit structure of the Web (seen as a labeled graph), for example STRUDEL [FFK+ 97] or WebOQL [AM98]; others, such as AltaVista, are based on a at analysis (through keywords) of textual content. However, the Web does not behave like a standard database. There is no super-user in charge of monitoring the source(s) (the data is constantly updated), there is no homogeneous structure (and hence no common explicit structure), the Web itself never stops growing, etc. For these reasons, we believe that the sourcedriven standard approach is not suitable for the Web. As an alternative, we propose a user-oriented approach. When a user formulates a query in a given language, the only structure (of the source) used to answer the query is that which is explicitly expressed in the query. Therefore, instead of extracting in any way a \super schema" for the Web, why not simply use the schema explicitly expressed by the user? In AKIRA (Agentive Knowledge-based Information Retrieval Architecture), we propose exactly this: the system uses a target structure inferred from the user's query.
1.2 Where and how to get the information The Web provides information (Web documents) as well as information about this information (search engines, etc.). The AKIRA system uses them both. The relevant information needed to answer the user's query is gathered from one or more Web documents. Two problems immediately arise: (1) where to nd the relevant documents? (2) how to understand their content? Useful services such as search engines, directories, etc. available on the Web help to solve the former, but return a lot of noise and redundancy. Their output must be ltered in order to select only relevant documents, a task related to problem (2). In AKIRA, we propose to process the content in order to locate the relevant documents and extract explicit or implicit structure from the retrieved documents. Our system has a exible architecture based on agents to access all these services.
1.3 Beyond traditional information processing The Web provides access to multimedia documents, but most of them rely partly or totally on textual content. Raw textual data is usually poorly analyzed through keywords: a \bag of words" representation of a document is often inadequate. Most Information Retrieval (IR) systems only index on words. A few systems use pattern matching or retrieval through phrases. However, these approaches do not capture any structure implicitly expressed in the text: no concept is extracted (for example, all names of persons may be indexed but no conceptual class Person is created), and no relationship (implicitly expressed by the order or position of words or sentences) is understood.
A User-Oriented Approach to Querying the Web
291
More advanced approaches could use syntactic or semantic tagging. Words could be annotated according to their part of speech categories (noun, verb, adjective, etc.) or grouped into categories such as Person to create concept classes. For instance, in the sentence \Barbara Pernici is the Program Chair for CAISE98", Barbara Pernici and Program Chair are respectively instances of concept classes Person and Designation. The next step would be to provide some relations between these concepts by introducing meta-concepts. Techniques from computational linguistics may be used for this purpose. In our example, classes Person and Designation are connected through the relation Appointment. SuperTagging [JS94], which provides rich syntactic labels, may be used with other tools (for example a co-reference tool [BDR+ 97]) to capture a variety of syntactic and semantic information, and thus meta-concepts such as Appointment. 1.4
Using DB techniques
The user-oriented paradigm means that the structure through which the data is viewed does not come from the source but is extracted from the user query. When a query is evaluated, the relevant documents are retrieved from the Web and stored as is. Then the information is extracted from the raw data using computational linguistic techniques. The AKIRA cache (smart-cache) stores these extracted layers of meta-information on top of the raw data. The smart-cache is an object-oriented database whose schema is inferred from the user's target structure. It is designed on demand out of a pool of conceptual schema components that can be assembled together to match concepts and meta-concepts required in the user's query. The smart cache can be seen as a view of the Web. The use of a database as a cache provides the system with a real query language PIQL (an extension of OQL [Ba97] with path expressions a la POQL [CCM96]) to query the Web. The paper is organized as follows. Section 2 presents an overview of currently available Web representations and query languages, and sketches an outline of AKIRA. Section 3 presents a motivating example, de nes the central concept of Web views and shows how AKIRA models some of the components required to evaluate a query. The architecture of the AKIRA system is described in Section 4. The last section contains our conclusions. 2
Web Query Languages
We believe that a high-level Web query language should be provided to the user. This should allow the user to specify exploration plans to generate automatic browsing and analysis of the content. Successive steps of the navigational exploration should be performed transparently by the system on behalf of the user. In this section, we rst analyze the limits of available languages, investigate how the content of Web documents should be accessed by the queries and propose our representation of the Web and our query language.
292 2.1
Z. Lacroix, A. Sahuguet, and R. Chandrasekar Web representations and languages
The expressive power of a query language re ects the expressiveness of the underlying representation of information. The Web consists of documents (text, picture, audio, video, etc.) with a HTTP address (a URL). Seen as a table [URL, content], the only language available to query the Web is the \type-fetch'n read" language (see Table 1). The user actually types a query (which is nothing but a URL) and gets back one or more documents, or nothing. Within this representation, neither the meaning of the URL nor its content is understood or accessed by this primitive language. The user has to provide both the reading and understanding. The \type-fetch'n read" language is nothing but browsing. Representation (data type, model and organization) Language rel table [URL, content] \type-fetch'n read" rel table [URL, content] FO, Datalog [AV97] and table [URL, label, URL] W3QL [KS95] rel table [URL, title, text, type, length, modif] hypergraph and table [URL, label, URL] WebSQL [MMM97] oo object=page attribute=label WebOQL [AM98] oo object=page attribute=HTML (or source-driven structure) Penelope [AMM97] hypermedia oo objects=instances of concepts attribute=meta-concepts PIQL Table 1.
Web Representations and Languages
Web documents can also be viewed as a set of graphs not (fully) connected in any regular manner. Thus there is no a priori way to search for something. There is no expectation that pages will be connected to any other page, nor that any page will have links outside the page. The only way to explore is to start from a given point (a valid URL, a bookmark, etc.) and navigate from page to page, accumulating addresses of new pages along with knowledge about the Web itself. The usual way to go through this hypergraph is browsing (choosing a hyperlink to jump to, by reading the content of the page) and repeating this process until satisfactory answers are reached. This process is non-deterministic since it is based on successive choices of hyperlinks that may lead to the expected information. More advanced technologies may enable automatic (and thus deterministic) browsing. Based on a representation matching the hypergraph (labels) and some additional structure extracted from the source (HTML tags in particular), the proposed query languages express browsing queries based on a thorough knowledge of the hypergraph (where the user must know a priori the labels!) [AV97,KS95,AM98] and the extracted structure [MMM97,AMM97]. These languages can express queries such as \retrieve all pages accessible from the
A User-Oriented Approach to Querying the Web
293
URL http://www.pianosa.cnuce.cnr.it/ with a label path expression ending with caise98" (a label path expression is a list of labels).
These languages are limited because their representation of the Web is not compatible with a high-level Web query language that would access the content of Web multimedia documents. Instead, the user must cope with reading everything to extract relevant information and discarding anything spurious.
2.2 Accessing the content
What kind of information is available on the Web? One can nd any information, from unstructured documents to highly structured ones (databases), from raw information to information about the Web itself (search engines). But none of the proposed languages associated with the representation of the Web as a hypergraph (see Table 1) really automatically accesses or understands all this information. Various tools have been successfully developed to index speci c media formats. Search engines such as Yahoo, AltaVista, Excite, Infoseek or HotBot for text, WebSeer or Lycos for images, give access to pre-computed indexes of Web sources. But is querying a search engine querying the Web? A user may use a search engine to query the Web; in this case, no knowledge of the hypergraph is required. A query to a search engine is nothing but a parameterized URL. For instance, looking in Yahoo's base for \CAiSE" corresponds to typing the following URL: http://search.yahoo.com/bin/search?p=CAiSE. We regard search engine querying as parameterized browsing through databases which are preprocessed and materialized views of the Web. These databases only represent a view of a portion of the Web. Only documents visited by spiders and indexed are returned by queries to search engines. Their visibility comes out by being connected to other visible parts of the Web, or by intentionally being registered. Moreover, some speci c (types of) sites are not indexed on purpose. It follows that querying a search engine is limited because (1) it restricts the query to data stored in a given database, and (2) the structure of the database only captures shallow information about the structure of the content of the document (via its indexes). AKIRA takes advantage of usual search engines as well as new technologies from computational linguistics.
2.3 A Smart-Cache
We focus on the idea of querying the Web on-the- y, which means that the Web is not preprocessed, nor is the query restricted to any stored subset of the Web. This is in contrast to [ACC+ 97,AMM97], where parts of the Web are preprocessed and stored in a database, for which the schema is de ned according to its sources. The resulting database is no longer linked to its Web sources (that may be updated meanwhile). Thus, a query in their system is evaluated against a closed-world database, and not against the Web. A cache usually stores HTML pages in a dummy le-system where manipulation is based on page names. AKIRA replaces the standard browser cache by
294
Z. Lacroix, A. Sahuguet, and R. Chandrasekar
a smart-cache, which is an object-oriented database. Its smart cache is empty and unstructured before any query. The schema is inferred from the user's query (target structure). Classes are populated with information extracted from documents retrieved from the Web. Most of the above representations (see Table 1) are page oriented to match the Web represented as an hypergraph. However, in order to take advantage of Information Extraction (IE) techniques to analyze the content of Web information and extract information with respect to a given structure, a ner granularity is required. We choose a at representation (as opposed to a tree-like representation) based on fragments (see Section 3.6). A fragment represents an indexed part of a multimedia Web document. A concept is represented as an abstract object class populated with instances referring to several co-referent fragments. Metaconcepts are non-valued attributes between concept classes. Our smart-cache is a Web view. We adopt PIQL (Path Identity Query Language), an enrichment of the standard object query language OQL [Ba97] with identity invention a la IQL [AK89] and with generalized path-expressions a la POQL [CCM96]. Our model of Web view with its query language provides a rich uni ed formalization of the Web, useful for future optimization. We motivate our work with a detailed example in the next section.
3 AKIRA's Web Views In this section, we illustrate AKIRA's approach with a motivating example. We assume the reader is familiar with the concepts of object-oriented databases as presented in [AHV95]. In AKIRA, a Web view is an object-oriented database instance. The object schema consists of classes, a class hierarchy (subclassing relationship), attributes and methods with their signatures. Classes are populated with object identi ers (oids). AKIRA's view mechanism is inspired by the view mechanism introduced in [dSDA94,LDB97].
3.1 Motivating example Researchers are usually interested in Calls for Papers (CFPs) in subject areas related to their research. While CFPs are widely disseminated, being published in journals, newsgroups, mailing-lists etc., researchers would appreciate a facility to nd out about relevant CFPs at any given time. Calls for Papers are usually text documents with some minimal implicit structure. Typically, they provide information about: { the name, venue and dates of the conference/seminar/workshop, { the list of topics of interest, { the contact people to whom submissions are to be made, { the last date for submissions, etc. The AKIRA approach may be used to identify conferences of interest to particular users, using a query such as expressed in Example 1.
A User-Oriented Approach to Querying the Web
3RRORIVFKHPD FRPSRQHQWV
XVHU¶VTXHU\
7KH :(%
6PDUW&DFKH
3,4/TXHU\
5HWULHYDO
Fig. 1.
295
TXHU\UHVXOW
([WUDFWLRQ
AKIRA's query processing.
Example 1. The user wants information about upcoming conferences such as query Q1: \Conferences about Information Systems with a submission deadline after July 31, 1998?"
As illustrated in Figure 1, we rst extract the concepts expressed by the user in his query (see Section 3.3 and Section 3.4). We describe how the relevant documents may be identi ed in Section 3.5 and how they are processed as explained in Section 3.6 to populate the concept classes in Section 3.7.
3.2 Pool of schema components Before further describing the evaluation of a query, it is important to emphasize the capabilities of our approach. When querying a database, a user is restricted to a given and frozen organization of information, enforced by the creator when designing the schema. Should the user send a request beyond the schema, he will be denied access to the expected information. The creator has imposed his view to the user. This is the limitation of the source-driven approach. Our user-oriented paradigm grants more exibility by allowing the user to design his own view on demand. There is no magic: the limits are transfered to the extraction capabilities (see Section 3.6) of the system. AKIRA's administrator is in charge of representing these capabilities at a conceptual level in terms of schema components, in a modular way. He provides the user with a pool of schema components that can be combined to specify the user's view. An IE tool capable of identifying conference names is represented by a concept class Conference with attribute name. Similarly, a date extraction tool corresponds to a class Date with attributes month, day and year. Each of these classes is a schema component by itself. A concept class can be specialized according
296
Z. Lacroix, A. Sahuguet, and R. Chandrasekar
to other extraction capabilities. For example, attribute topic, listing topics of interest, can specialize class Conference. Two concept classes can be also combined through a meta-concept such as submission deadline to assemble a new conceptual schema as illustrated in Figure 2.
Conference name topic
Fig. 2.
3.3
Date
submission_deadline
year day month
Conceptual schema.
Query processing
AKIRA's framework is compatible with a Natural Language (NL) user interface such as described in [ART95]. In pattern-matching systems, a relational table is assumed and natural language patterns are associated with action rules. Similarly, in AKIRA, action rules can correspond to concept classes, as illustrated below. pattern: ... "conference name" ... action: select c.name from c in Conference
Action rules built for each schema component permit us to process query Q1 and obtain a corresponding PIQL expression: select c.name from c in Conference where "Information Systems" in c.topic and c.submission_deadline.month > 7 and c.submission_deadline.year = 1998
The NL interface matches the words expressed in the NL query with the user's target structure (see Figure 2). A standard NL database interface does not require the user to know the organization (schema) of data. Therefore AKIRA also provides action rules that translate patterns into generalized path expression a la POQL [CCM96]. Suppose that a user wants to know \Conferences where the temperature is over 90F". There is no built-in attribute temperature available for class Conference in the pool of schema components, however the system will translate the query using the pattern: ... "conference name" ... [temperature] associated with the action: select c.name from c in Conference where c.[*].temperature>90
A User-Oriented Approach to Querying the Web
297
where c.[*].temperature>90 is a general path expression. If attribute country is available at Conference and attribute temperature at class Country, then the system will infer from: select c.name from c in Conference where c.[*].temperature>90
the OQL expression: select c.name from c in Conference where c.country.temperature>90
3.4 View mechanism Our view mechanism goes through a pipeline (see Figure 3) of successive and interleaved views (obtained by successive materialized extensions [dSDA94,LDB97]). The main task consists in specifying the schema transformation from the current schema to the target structure. When the rst query is asked, the current schema is empty. In case of a re nement, a current schema (de ned to answer previous queries) structures the cache and has to be extended to support the target structure (by adding new classes and/or attributes). The input of the view mechanism is a PIQL query together with the current schema of the cache (if any). First the target structure has to be inferred from the PIQL query. In particular, the system has to resolve the general path expression (if any) by introspecting its pool of schema components for all possible matching paths. The view speci cation is derived as the di erence between the target structure and the current schema. The view mechanism forwards three queries to structure and populate the cache: 1. a query invoking IR tools to retrieve relevant documents from the Web, 2. a schema transformation query de ning the new structure of the cache (new classes and/or attributes) according to the user's target structure, and 3. an update query triggering methods that invoke IE tools to populate the cache using the content of retrieved documents.
3.5 Retrieving Relevant Information To answer Q1, we need to populate the cache, namely to identify pertinent CFP
information through the following steps. Information Retrieval: we can look for documents indexed by search engines which satisfy a query expressed as a boolean expression on keywords/phrases such as: "Calls for Papers" OR "Call for Papers". We can also use websites and newsgroups which collate information about conferences in one or more subject areas. For example, one can nd a list of CFPs about WWW, Hypertext, Structured Documents, Information Management, Authoring/Reading Tools, Reuse of Web Information, metadata, etc. at the URL
298
Z. Lacroix, A. Sahuguet, and R. Chandrasekar
These (typically volunteer) e orts are quite useful, but not always up to date, and not expected to be exhaustive in any way. For a variety of reasons, the AKIRA approach is likely to be signi cantly better than using such repositories per se. Information Filtering: in a second step, we discard documents which are likely to be spurious or lacking in relevant information. These include documents which do not contain the standard CFP phrases, documents which are Web-redirection documents, empty documents etc. We may also discard documents which contain the word Archive (these may be mega-Archives without much relevant content). A ltering tool such as Glean [CS97] may be used for this purpose. http://www.mel.dit.csiro.au:8080/ delloro/db/.
3.6 Extracting Information: fragments From retrieved documents, we identify names of meeting and dates thanks to our IE agents. A conference is identi ed by its name and has a canonical representation expressed by an acronym (for example CAISE98). A date is a string of characters expressing a month, a day and a year. Its canonical representation (aka normalized representation) is a list of three integers (for example [11,30,1997]). We introduce the notion of fragment, which is a exible way to consider a document in di erent granularities, according to the needs of the user. Fragments correspond to the strings of characters indexed by IE agents in the retrieved documents as illustrated in Figure 3. Each fragment is characterized by a pair consisting of a document name and a span (a span consists in turn of a pair of integers specifying the starting and ending position of the indexed string of characters in the document [Gri96]). When the fragmentation is accomplished, concept classes may be populated. RULJLQDO GRFXPHQW
QHZLQVWDQFHV RIFODVV&RQIHUHQFH ³&RQIHUHQFH´
³'DWH´
$JHQW
$JHQW
([WHQVLRQ 3KDVH Fig. 3.
AKIRA's fragmentation pipeline.
QHZLQVWDQFHV RIFODVV'DWH
A User-Oriented Approach to Querying the Web 3.7
299
Concept classes
As explained in Section 3.4, the target structure inferred from query Q1 speci es the object schema of the smart-cache as follows. Class Conference { oid fragments name submission_deadline topic Class Date
Each extracted conference name is represented by its canonical form. For instance, fragments such as CAISE, 10th Conference on Advanced Information Systems Engineering, etc., are represented as an object instance of class Conference. The value of its attribute name is its canonical representation CAISE98, and the value of its attribute fragments, the set of all fragments it refers to. Class Date is similarly populated. The value of extra attributes such as topic is extracted by an IE tool (for example, a zoner1 that extract zones mentioned as \Conference Topics", etc.) from CFPs. For each instance of a conference, the value of attribute topic is the set of all topics extracted from the \Conference Topics" zone of its CFP. Meta-concepts such as submission deadline also invoke IE tools to extract the relationship between two concepts. For example, from the CFP of a conference, a zone \Important Dates" can be identi ed from which the submission deadline can be extracted. Another tool may exploit SuperTagging [JS94]. A training phase consists in extracting patterns from the sentences where the submission deadline is expressed (such as \All submissions must be sent to the PC chair by November 11, 1997" or \Authors are invited to submit a position paper no later than November 11, 1997", etc.) in a sample of CFPs. The extraction phase consists in (1) retrieving sentences where \send", \submit", etc. occur (with a grep) and comparing their pattern with the ones obtained from the training session; and (2) extracting the date from each sentence that matches a pattern and identifying the conference submission deadline.
4 AKIRA architecture The AKIRA system can be viewed as a personal proxy that provides the user with transparent access to the Web: the input to AKIRA is provided through a 1 See [AI97] for zoning extraction tools.
300
Z. Lacroix, A. Sahuguet, and R. Chandrasekar
standard HTML form or through a parameterized URL while the output is an HTML page generated on-the- y by the system. These \virtual" pages, similar to the virtual documents in [VDH97], can be bookmarked and reconstructed on-demand. The AKIRA system basically receives a query, creates an object-oriented database (a Web view), and returns the output of the query against the instance of the database. It has ve components: the Dispatcher, the DBMS (DataBase Management System), the View Factory, the Agent Pool, and the Output Formatter as illustrated in Figure 4. XVHU¶VTXHU\
'LVSDWFKHU
$ * ( 1 7 3 2 2 /
2XWSXW )RUPDWWHU
9LHZ )DFWRU\
XVHU¶VUHVXOW
'DWDEDVH
6HUYLFHV ,5IRUPDWHWF
Fig. 4.
AKIRA's architecture.
The Dispatcher has a role similar to the one of a query processor for a database management system. It translates the user's query in a PIQL expression and extracts the target structure. The View Factory is an essential part of the system. The View Factory's task is to populate the cache with information extracted from documents retrieved from the Web by IR agents. The Database System (DBMS) storing the expected Web view is objectoriented. It is de ned with a view expression sent by the View Factory which speci es its schema as well as its population. The Agent Pool contains IR, IE, formatter agents, etc. IR agents consist of wrappers to correspond with data sources available on the Web (search engines or services), and information ltering tools such as Glean [CS97]. IE agents extract concepts and meta-concepts. IE agents such as conference acronym and location recognizers together with a co-reference tool identify concept instances. SuperTagging [JS94], which provides rich syntactic labels, and zoners extract
A User-Oriented Approach to Querying the Web
301
meta-concepts. Formatter agents can be of type summarizer, table-of-content, glossary, etc. The Output Formatter, is used to format the output according to the user's needs. The motivating CFP example provides only a glimpse of the range of capabilities of the AKIRA system. 5
Conclusion
In this paper, we have described AKIRA, an alternative approach to querying the Web. Here are some of the several bene ts to using the AKIRA framework: 1. Bene ts from Natural Language techniques: Techniques from natural language processing provide access to explicit as well as implicit structure of textual content. Some of the ideas we are discussing have been proposed in other contexts (for example, [SLE97]). 2. Bene ts from Database techniques: The separation between the logical view (concept and meta-concepts) of Web documents and its storage in the smart-cache presents several advantages, including a Web query language. Its schema is tailored by the user when asking a query. Our approach does not require the integration of several heterogeneous sources in a global common representation. Moreover, it is worth noting that AKIRA does not assume that it can start from a database representation (schema and instances) of the Web like many other systems dealing with site-restructuring (see for instance [FFLS97,AM98,GW97]). 3. Bene ts from the AKIRA architecture: AKIRA o ers a transparent architecture to access data of various media from the most loosely structured sources (newswire, press release, personal homepages or newsgroups) to highly structured sources (legacy databases, catalogs, digital libraries). Its modular framework and extensible design provides the user with a highly tunable interface to the Web. We present two important directions for future work.
Understanding hypermedia documents: Web documents are multimedia
and our conceptual representation is medium-independent. AKIRA will take advantage of various tools successfully developed to index speci c media formats. IE tools usually parse linear textual documents. They should rst be generalized to mark-up language syntax (SGML, HTML, XML, etc.) in order to understand and use the meta-organization provided by tags. Moreover, a Web document is no longer a single linear page but a hyperdocument (a graph of connected nodes). IE tools should be able to extract structure from a hyperdocument and thus over hyperlinks. AKIRA's approach aims at automating browsing. When IE tools can adjust the hyperstructure of Web documents, heuristics can be introduced to select hyperlinks according to a strategy which may be used to mimic human browsing.
302
Z. Lacroix, A. Sahuguet, and R. Chandrasekar
AKIRA can take advantage of knowledge representation. For instance, by using a topic hierarchy and a thesaurus, AKIRA can be programmed to retrieve information about particular subject areas and all its super-areas. An approach combining knowledge representation and natural language processing such as conceptual indexing [Woo97] could dramatically improve AKIRA's ability to retrieve relevant information. Quality of service: AKIRA's system is subject to the inherent hazards of information processing techniques (recall/precision). However, it aims at delivering information together with a measure of con dence. Our deliberate choice of processing data on-the- y forces us to emphasize the issue of performance. Standard database query rewriting can be considered to optimize the evaluation of the query on the database instance [CCM96]. The view mechanism itself may be tuned according to both the view de nition and the retrieval of documents. Other approaches to manage semi-structured data such as Lorel [AQM+ 97] could be investigated. The AKIRA system [LSC98] is under development at the Institute for Research in Cognitive Science in collaboration with the Database group of the University of Pennsylvania.
Acknowledgment: Alberto Mendelzon and Anne-Marie Vercoustre are thanked for valuable comments on an earlier version of the paper.
References
[ACC+ 97] S. Abiteboul, S. Cluet, V. Christophides, T. Milo, G. Moerkotte, and J. Simeon. Querying documents in object databases. Journal on Digital Libraries, 1997. [AHV95] S. Abiteboul, R. Hull, and V. Vianu. Foundations of Databases. Addison{ Wesley, 1995. [AI97] D.E. Appelt and D. Israel. Building information extraction systems. In ANLP-97 Tutorial, Washington, D.C., March 1997. [AK89] S. Abiteboul and P. Kanellakis. Object Identity As A Query Language Primitive. In ACM SIGMOD Symposium on the management of Data, pages 159{173, Portland Oregon USA, June 1989. [AM98] G. Arocena and A. Mendelzon. WebOQL: Restructuring Documents, Databases and Webs. In Proceedings of the International Conference on Data Engineering, Orlando, February 1998. [AMM97] P. Atzeni, G. Mecca, and P. Merialdo. To Weave the Web. In Proc. of Intl. Conf. on Very Large Data Bases, Athens, Greece, August 1997. [AQM+ 97] S. Abiteboul, D. Quass, J. McHugh, J. Widom, and J.L. Wiener. The Lorel Query Language for Semistructured Data. Journal on Digital Libraries, 1997. ftp://db.stanford.edu/pub/papers/lorel96.ps. [ART95] I. Androutsopoulos, G.D. Ritchie, and P. Thanisch. Natural language interfaces to databases - an introduction. Journal of Natural Language Engineering, 1(1):29{81, 1995. Cambridge University Press. http://www.mri.mq.edu.au/ ion/nldbsurv.ps.gz.
A User-Oriented Approach to Querying the Web
303
[ART97] I. Androutsopoulos, G.D. Ritchie, and P. Thanisch. A framework for natural language interfaces to temporal databases. In Proceedings of the 20th Australasian Computer Science Conference, volume 19(1), pages 307{315, Sydney, Australia, 1997. Australian Computer Science Communications. http://www.mri.mq.edu.au/ ion/acsc97.ps.gz. [AV97] S. Abiteboul and V. Vianu. Regular Path Queries with Constraints. In Proc. ACM Symp. on Principles of Database Systems, 1997. [Ba97] D. Bartels and al. The Object Database Standard: ODMG 2.0. Morgan Kaufmann, San Francisco, 1997. [BDR+ 97] B. Baldwin, C. Doran, J.C. Reynar, B. Srinivas, M. Niv, and M. Wasson. EAGLE: An Extensible Architecture for General Linguistic Engineering. In In Proceedings of RIAO'97, Montreal, June 1997. [CCM96] V. Christophides, S. Cluet, and G. Moerkotte. Evaluating Queries with Generalized Path Expressions. In Proc. ACM SIGMOD Symp. on the Management of Data, 1996. [CS97] R. Chandrasekar and B. Srinivas. Using Syntactic Information in Document Filtering: A Comparative Study of Part-of-speech Tagging and Supertagging. In In Proceedings of RIAO'97, Montreal, June 1997. [dSDA94] C. Souza dos Santos, C. Delobel, and S. Abiteboul. Virtual Schemas and Bases. In Proceedings of the International Conference on Extending Database Technology, March 1994. [FFK+ 97] M. Fernandez, D. Florescu, J. Kang, A. Levy, and D. Suciu. STRUDEL: A Web-site Management System. In ACM SIGMOD { Research prototype demonstration, Tucson, Arizona, May 1997. [FFLS97] M. Fernandez, D. Florescu, A. Levy, and D. Suciu. A Query Language and Processor for a Web-Site Management System. In ACM SIGMOD Workshop on Management of Semistructured Data, Tucson, Arizona, May 1997. [Gri96] R. Grishman. TIPSTER Text Phase II Architecture Design. Technical report, TIPSTER Text Program, 1996. http://www.tipster.org/docs/arch23.ps.gz. [GW97] R. Goldman and J. Widom. DataGuides: Enabling Query Formulation and Optimization in Semistructured Databases. In Proc. of Intl. Conf. on Very Large Data Bases, Delphi, Greece, August 1997. to appear. [JS94] A.K. Joshi and B. Srinivas. Disambiguation of Super Parts of Speech (or Supertags): Almost Parsing. In Proceedings of the 17th International Conference on Computational Linguistics (COLING '94), Kyoto, Japan, August 1994. [KS95] D. Konopnicki and O. Shmueli. W3QL; A query system for the World Wide Web. In Proc. of Intl. Conf. on Very Large Data Bases, 1995. [LDB97] Z. Lacroix, C. Delobel, and Ph. Breche. Object Views and Database Restructuring. In Proc. of Intl. Workshop on Database Programming Languages, August 1997. [LSC98] Z. Lacroix, A. Sahuguet, and R. Chandrasekar. User-oriented smart-cache for the Web: What You Seek is What You Get! In ACM SIGMOD { Research prototype demonstration, Seattle, Washington, USA, June 1998. http://www.cis.upenn.edu/AKIRA. [MMM97] A. Mendelzon, G. Mihaila, and T. Milo. Querying the World Wide Web. Journal on Digital Libraries, 1(1):54{67, 1997. [RC93] S. Ramani and R. Chandrasekar. Glean: a tool for Automated Information Acquisition and Maintenance. Technical report, NCST Bombay, 1993.
304
Z. Lacroix, A. Sahuguet, and R. Chandrasekar
[SLE97]
J. Shakes, M. Langheinrich, and O. Etzioni. Dynamic reference sifting: A case study in the homepage domain. In Proceedings of the Sixth International World Wide Web Conference, pp.189-200, 1997), 1997. [VDH97] A-M. Vercoustre, J. Dell'Oro, and B. Hills. Reuse of Information through virtual documents. In Proceedings of the 2nd Australian Document Computing Symposium, Melbourne, Australia, April 1997. [Woo97] W.A. Woods. Conceptual indexing: A better way to organize knowledge. Technical Report TR-97-61, Sun Microsystems Laboratories, April 1997.
Goal-Driven Business Process Analysis Application in Electricity Deregulation V. Kavakli and P. Loucopoulos Department of Computation U.M.I.S.T. PO Box 88, M60 1QD, Manchester, UK {kavakli | pl} @co.umist.ac.uk
Abstract Current business challenges such as deregulation, mergers, globalisation and increased competition have given rise to a new process-centric philosophy of business management. The key issue in this paradigm is the concept of business process. From a methodological perspective, this movement has resulted in a considerable number of approaches that encourage the modelling of business processes as a key component of any improvement or reengineering endeavour. However, there is a considerable controversy amongst all these competing approaches about the most appropriate way for identifying the types and number of relevant processes. Existing business process modelling approaches describe an enterprise in terms of activities and tasks without offering sufficient guidance towards a process-centred description of the organisation. In this paper we advocate the use of a goal-driven approach to business process modelling. A systematic approach to developing and documenting business processes on the basis of the explicit or implicit business objectives is put forward. We argue that such an approach should lead to a closer alignment between the intentional and operational aspects of an organisation. Our approach is exemplified through the use of parts of a large industrial application that is currently making use of a goal-driven business process modelling.
1
Introduction
The traditional practice of managing an enterprise adopts a functional view in which the business is organised along individual types of work performed, resulting in organisational structures which reflect the particular functional view adopted by the business. The main reason for adopting a functional organisation is the achievement of maximum performance of individuals or business functions. Nevertheless, this inward focus on ‘internal’ performance rather than ‘global’ efficiency suffers from a number of drawbacks, especially when business improvement is sought. In particular, improvements occur piecemeal and independently of one another, while concentration on the symptoms of one function ignores causes in important crossfunctional interdependencies. B. Pernici and C. Thanos (Eds.): CAiSE’98, LNCS 1413, pp. 305-324, 1998. Springer-Verlag Berlin Heidelberg 1998
306
V. Kavakli and P. Loucopoulos
Current business challenges such as deregulation, mergers, globalisation and increased competition, have given rise to a new philosophy of business management that organises an enterprise in terms of processes rather than functions and tasks. The basic characteristic of this approach is the re-orientation of business from performing as a cluster of functions or divisions to integrating activities within a limited number of core processes. Each core process captures cross-functional interdependencies and concentrates on few strategic objectives that determine competitive success. Therefore, a process centred approach links improvement efforts in different functions to a shared set of strategic objectives. Adopting a process view however, requires suitable tools for identifying, modelling and measuring business processes. Existing business modelling approaches describe enterprises in terms of activities and tasks offering little or no guidance towards a process-centred description of the organisation. In this paper we advocate the use of a goal-driven approach whereby a business is seen as a purposeful system aiming to achieve defined objectives which add value to its customers. This approach is part of a larger enterprise knowledge modelling framework, known as the EKD approach [Loucopoulos, Kavakli, et al 1997]. Allied to business process modelling is the larger issue of business change itself. Business change is also seen as goal-driven in EKD; the need for business change is externalised in terms of strategic business goals, which in turn shape business processes. Therefore, business change management is the process of identifying the business goals for change and analysing the impact that these goals have to business processes. The paper is organised as follows. Section 2 introduces the industrial application which is referred to throughout the paper. Section 3 introduces the notion of business process in terms of its defining characteristics and presents a critique of existing process modelling techniques. Section 4 briefly introduces the goal-driven approach to business process modelling. The application of the approach is illustrated in section 5, using examples from the industrial application introduced in section 2. Finally, section 6 concludes with a discussion on the role of goal-driven business process modelling within the broader context of business change management.
2
Background to the Application
The work presented in this paper is part of a big industrial application that concerns de-regulation of a large European electricity company. The company is divided in three operational areas generation, transmission and distribution. Generation is responsible for the production of electrical power. Transmission is responsible for the high voltage transport of electricity. Finally, distribution is responsible for the medium voltage (M/V) and low voltage (L/V) transport of electricity, its delivery to consumers and the merchandising of electricity
Goal-Driven Business Process Analysis - Application in Electricity Deregulation
307
services. These areas operate under the rules and regulations of a governmental regulatory body that controls issues like tariffs, production levels, environmental policies, etc. Currently the company operates in a total monopoly market which means that it is the single operator of all three areas. A high-level view of the main company actors and their roles is illustrated in Fig. 1. Customer
Generation Operator
Transmission Operator
Electricity Generation
Distributor
Buying Electricity
Transmission Distribution
C
Operate transmission network
Produce electrical power
R
Supply electricity C to customer
R
Regulator
Buy electricity
Regulation
R
A A A
Regulate electricity market
Fig. 1. Main company actors and their roles in the monopoly market
In anticipation of the opening of the European electricity market, the company is in the process of re-designing its business structure and planning reforms for the future, in order to increase its competitiveness and retain its market share. This is especially critical in the distribution area which is the interface of the company with the final customer. Adopting a process view of the business is a key factor in this effort. Experience from previous projects in the company has shown the need for a structured approach for describing and measuring the business processes. Nevertheless current methods focus on what it is done (the tasks and activities performed) rather than how work is done in terms of processes, offering little assistance in this direction. This study reports on the application of a goal-driven approach whereby business goals are put forward while identification and analysis of business processes is based on their intentional affinity. For the purpose of this paper we focus on one section of the distribution area, namely the Distribution District. The current structure of a Distribution District is organised along four distinct functional sections illustrated in Fig. 2: the Technical Section, the Customer
308
V. Kavakli and P. Loucopoulos
Electrification Section the Personnel Section and the Customer Services Section (or agencies). D istrict
T e ch nic al Se c tion
C u sto m er E lec trific ation S e c tion
P e rson n e l Se c tion
C u sto m er S e rv ic es S e c tion
Fig. 2. Functional organisation of a District
The Personnel Section deals with internal matters of District employees, including safety and training issues. The Customer electrification section mainly plays a manager role. It is responsible for checking and checking all expenditures and authorising the construction of works that concern the electrification of customers as well as the managing of customer payments to the company. The executive roles are played by the Technical Section. The Technical Section is responsible for the operation and maintenance of the distribution network, as well as the technical servicing and maintenance of customer installations. Finally the Customer Services Section plays mainly administrative roles being the interface between the electricity consumer and the District. In addition the customer services section performs periodical readings of the electricity metering devices at customer installations in order to calculate electricity consumption and receives customer payments.
3
Business Process Modelling
The concept of business process is a key issue in the process centred paradigm. However, there is a considerable controversy around the number and types of processes appropriate to a given organisation [Davenport 1993]. The difficulty derives from the fact that there exists no explicit way for determining business processes. There is a lack of a coherent and universally accepted definition of what a business process actually is. Nevertheless, there are some common features of business process definition in the literature [Alderman, Maffin, et al 1997; Davenport 1993; Hammer and Champy 1993; Ould 1995] that provide guidance as to how business processes should be defined. In summary a business process in the process-centred organisation demonstrates the following characteristics: •
a business process has well identified products and customers, such that business objectives are matched through the (product offering) business process and delivered in the form of the product; customers may be external or internal to the organisation; products may include finished goods or services
Goal-Driven Business Process Analysis - Application in Electricity Deregulation • • •
309
a business process has goals, i.e., it is intended to achieve defined business objectives aiming to create value to customers a business process involves several activities which collectively achieve defined business process goals and create value to customers a business process crosses functional/organisational boundaries; it concerns the collaboration between organisational actors that are contributing to (or constraining) the satisfycing of business objectives
In these terms a business process constitutes the manifestation of what organisational actors do in order to achieve business objectives. Organisational actors include individuals or groups which may be internal or external to the organisation (e.g., company employees, organisational departments, customers, suppliers etc.) and influence the realisation of business objectives. Business objectives aim at creating value to customers in other words they concern customer value goals. Business process modelling is a generic name that refers to a collection of techniques which are used to model the behaviour of business systems. Existing process modelling approaches mainly originate from the software engineering field and fall in one of three categories: •
•
•
Activity-oriented approaches describe a process as a set of ordered activities (e.g., SADT [Ross and Schoman 1977], IDEF0 [IDEF0 1993], DFDs [DeMarco 1978], Workflows [Swenson and Irwin 1995], the F3 process model [Bubenko 1994]). The emphasis is on what activities take place. Each of these activities is decomposed in smaller tasks corresponding to smaller steps in the process. In addition to a collection of tasks activity-oriented models define the order of task invocation or condition(s) under which tasks must be invoked, task synchronisation, and information flow. Agent-oriented (or role-oriented) approaches specify and analyse the role of the agents that participate in the process (e.g., Role Interaction Nets [Singh and Rein 1992], Role Activity Diagrams [Ould 1995], the i* model [Yu 1994], the ORDIT approach [Dobson, Blyth, et al 1994]). The focus is on the entity that performs a process element. Roles represent the sequences of activities carried out by agents engaged in a co-operative behaviour. Product-oriented approaches represent a process through the evolution of its products (e.g., [Easterbrook and Nuseibeh 1995], [Franckson and Peugeot 1991]). Product oriented models do not put forward the activities involved in a process but rather the result of these activities. The focus is on products and transformations made on them. Each product entity has a defined sequence of states and triggers that cause state transformations.
All the above approaches promote a view of a process that is based on the notion of activity. Activity-oriented approaches focus solely on description of activities. In addition product-oriented approaches couple activities to their output (the product),
310
V. Kavakli and P. Loucopoulos
while agent-oriented approaches establish an explicit link between the activities and the agent responsible for these activities. Existing approaches offer little guidance for identifying business processes. In activity-oriented approaches the main mechanism for grouping activities into processes is that of composition/de-composition. This mechanism however, does not offer a unique way to identify a process. The difficulty derives from the fact that processes are almost indefinitely divisible; the activities involved in fulfilling a customer order, for example, can be viewed as one process or hundreds. Agentoriented approaches on the other hand, group activities into processes according to the organisational agent that performs these activities. Yet, a process may cut across the organisation involving several organisational agents. Finally, product-oriented approaches group activities based on the product that they manipulate and this notion of a process is in accordance with the suggested business process definition as the delivering of products to customers. However this focus on product rather than organisational behaviour fails to describe other important components of a business process such as the business goals that the process intends to achieve and the collaboration of the agents that contribute to the realisation of process goals.
4 4.1
The EKD Approach to Business Process Modelling Overview
It becomes obvious that taking a single modelling perspective (product, activity or role) is not sufficient for expressing business processes. A different approach towards business process modelling is taken in the EKD approach promoted in [Loucopoulos, Kavakli, et al 1997]. In this view, EKD is a systematic approach to developing and documenting enterprise knowledge, helping enterprises to consciously develop schemes for implementing changes. EKD advocates a goal oriented view to business process modelling. Instead of imposing a single modelling criterion EKD offers a more general modelling framework that allows several modelling views (or rather modelling components), using the notion of business goals to structure business components in coherent business processes. The above are summarised in Fig. 3 which presents an overview of the EKD modelling concepts. In more detail, a business enterprise in EKD is described as a network of related business processes which collaboratively realise business goals. Business processes are supported by business systems. In the District example the ‘customer electrification’ process, realises the business goal ‘satisfy customer demand for electricity’ and is supported by the ‘customer information system’. Business processes are composed of roles that actors (individuals or groups) play in order to meet their responsibilities. An actor is the physical entity (e.g., the
Goal-Driven Business Process Analysis - Application in Electricity Deregulation
311
‘District technician’,
or the ‘District Technical Section’) that plays one or more roles. A role expresses a collection of responsibilities (e.g., ‘service providing’, ‘service administrative handling’, etc.) and involves a set of activities. For example the ‘service providing’ role involves activities such as, ‘construct customer installation’, ‘install metering device’ and ‘connect meter to the electricity network’).
Business Goals Business Objectives
realised_by Objects Actors Business Processes
Roles
Rules Activities
supported_by Information Systems Business Systems
Fig. 3. Overview of EKD modelling components
Activities carried out by different roles deal with business objects; business objects are manipulated by business activities and define the resources and information necessary in order to support the way that enterprise actors fulfil their role. For example the ‘installation’ object is the result of the ‘construct customer installation’ activity and is described by the following information in the ‘customer information system’: installation number, service start date, address of installation, town, town code, owner’s name and building location. Finally, business processes take place according to a particular logic (or business rules); business rules determine the allowable states of business objects and determine the interactions between different roles. An example of a business rule concerning the installation object is ‘WHEN application form submitted IF contract = signed THEN authorise construction of customer installation’. 4.2
Goal-Driven Business Process Modelling
An important aspect of business process modelling in EKD is the representation of business goals. Indeed business processes constitute the means to fulfil strategic business goals. A business process is also seen as a purposeful system in itself. Each role involved in the process intends to achieve one or more defined goals. This does
312
V. Kavakli and P. Loucopoulos
not necessarily mean that every role in a process aims to achieve the same business goal rather that satisfaction of the ‘private’ goals of individual roles supports the achievement of the business goal that is realised by the business process. Therefore, goals related to a business process present a hierarchical structure whereby individual role goals constitute refinements of higher-level goals that ultimately make up the business goal fulfilled by that business process (see Fig. 4). In this sense business goals not only define but also shape business processes. realised_by
Business Process: Customer Electrification
G0 Actor 1 G1,1
G1,2 Role 1
Gi,1 Gi,1
Gi,2 ... Gi,j
Gi,j+1 ... Gi,n
Actor 2 Role 2
Actor n
Gi,2
Role n
. . .
Gi,j
Gi,j+1 . . .
Gi,n . . .
Fig. 4. Relation between business goals and business processes
In the example illustrated in see Fig. 4, Role1: ‘service providing’ role achieves goal Gi,1:‘construct new customer installation and connect it to the electricity network’. On the other hand Role2:‘service administrative handling’ role achieves many goals one of which is the goal Gi,2:‘administer servicing of customer’s request for electricity’. Achievement of both goals supports achievement of the overall business goal G0:’satisfy customer demand for electricity’ which is realised by the ‘customer electrification’ process. Thus ‘service administrative handling’ and ‘service providing’ roles form part of the ‘customer electrification’ process. Business goals do not just shape the current business structure. They also set the vision for business change or business improvement. To this end, business goals establish the context of business change (i.e. the objectives towards which the business change effort is targeted). For example the business goal ‘increase District competitiveness’ sets the context of business change for the District case. Achieving this goal can be seen as a gradual process which encompasses the causal transformation of the initial goal into one or more subgoals until a plausible business process specification that satisfies the original goal has been defined. In our example the original goal ‘increase District competitiveness’ can be refined in the subgoals ‘create new markets’, ‘build a commercial profile’
Goal-Driven Business Process Analysis - Application in Electricity Deregulation
and
313
‘improve current functioning’.
The latter can be consecutively refined into and ‘reduce response time of any customer request’. This is graphically represented in Fig. 5. Any goal at each refinement level describes WHAT needs be done. At the same time this goal can also be considered as an end (WHY) for another goal, as well as means (HOW) for still another goal at a higher level. ‘improve existing services to current customers’
Intentional features
Increase District competitiveness
Goal
WHY
Legend
AND decomposition
Create new markets
Build a commercial profile
Improve current functioning
Improve product quality
Operational features
WHAT
Improve existing services to current customers
WHY
HOW
WHAT
Reduce response time of any customer request
HOW
Fig. 5. Business goals define the context of business change
In many cases more than one alternative subgoals can be identified. This will lead to the identification of alternative ways to achieve a business goal and therefore alternative ways of shaping the business. We must note here that goal achievement is not a strict top-down refinement sequence. One could also proceed bottom-up by finding simple goals and then connecting them to higher level ones. Of course, the initial change goals are defined first – otherwise there would be no subject-matter for the whole process.
5 5.1
Applying Goal-Driven Business Process Modelling Relate Business Goal Satisfycing to Process Modelling Strategy
In this section we discuss the empirical results and observations from applying the approach briefly discussed in section 4, to the industrial application (introduced in section 2). Any design task for change normally involves multiple stakeholders and decision makers. One of the aspects of the EKD approach is the support of a reasoning cycle that involves goal setting, deliberation and agreement. Space limitations prevent us from giving a full treatment to this subject but, since it is relevant to the business process modelling activity we briefly describe its use with reference to the industrial application.
314
•
•
•
V. Kavakli and P. Loucopoulos
Goal setting consists of establishing the stakeholder goals which designate any objectives to be reached, demand to be satisfied, problem to be resolved, issue to be discussed, etc. in general anything that one would like to achieve in using EKD. Deliberation includes the expression of hypotheses for achieving stakeholder goals (e.g., expressing alternative problem resolutions, making proposals concerning the satisfaction of some demand, etc.) as well as generating arguments for or against such hypotheses. Finally, agreement generates decisions that can alter (produce/modify) the product (the EKD models) while in turn generate new goals to be achieved. GOAL
Re-organise District to handle competition ARGUMENT
re-organisation requires a clear view of where the business currently stands GOAL
Describe the current District situation adopting a processcentred perspective
ARGUMENT
re-organisation requires a clear vision of where the business wishes to be in the future
ARGUMENT
a business process is the manifestation of how business actors co-operate to achieve business goals DECISION
Identify current business actors, roles and their interrelationships DECISION
Identify current business goals GOAL
Analyse District processes in the context of change goals
ARGUMENT
business process re-design is the operationalisation of business goals for change in terms of business processes
DECISION
‘Re-focus’ business roles towards business processes based on the goals they are trying to achieve DECISION
Identify District objectives for change
DECISION
Relate goals for change to existing business processes DECISION
Use business goals for change to identify criteria for re-designing related business processes
Fig. 6. Reasoning in the District application
Goal-Driven Business Process Analysis - Application in Electricity Deregulation
315
The benefit from using such an approach is twofold. First, the important components of any deliberation are captured and can be used in tracing the history of the rationale of decisions. Second, it can be used as the baseline for evaluating these decisions, relating these to the business goals, business processes and support systems and acting as a framework for quantifying design options. Application of EKD in the District case involved several technical and managerial staff together with EKD experts. An example of the reasoning cycle applied to the way that we approached the business process modelling task is shown in Fig. 6. 5.2 Model District Organisation
Micro-Processes according
to Current
Functional
A summary of the business activities performed in each functional section is presented in Fig. 7 which presents a map of District activities as described by the District employees. This map represents a ‘vertical’ view of the District in which District activities (or rather micro-processes) are organised along the four functional lines introduced in Fig. 2. Technical Section
Customer Services Section A1 - Electricity Supply Application Fulfillment for L/V Customers A2 - Network Resitation A3 - Meter Disconnection A4 - Meter Re-connection A5 - Meter Check & Maintenance A6 - Installation Modification A7 - Failure Restoration Customer A8 - Grant Bank Payment Authorisation Services A9 - Revoke Bank Payment Authorisation A Section A10 - Billing Correction (Agencies) A11 - Payment Collection A12 - Meter Reading
D1 - Personnel Training D2 - Prevention of industrial accidents
DISTRICT
B1 - Handling Damages caused on PPC’s Networks by third parties B2 - Performing Study of Electricity Supply through Modification of L/V Network B3 - Performing Study for New or Existing M/V Customers B4 - B5 - Performing Study of Electricity Supply without Modification of L/V Network B5 - Performing Study on Network Modification B6 - Repair of Damages to Network due to Natural Disasters or Black out B7 - Electricity Disconnection B8 - Electricity Re-connection B9 - Network Modification B10 - New 20/0.4 KV Substations Construction B11 - New U/G 20 KV Line (non Attica) Construction B12 - New Building Construction for 20/0.4 KV S/S B13 - U/G Cable Re-routing for 20KV Lines B14 - Drafting of provisions for a completed program B15 - Electricity failure repair by shifts B16 - Drafting of monthly electricity failures repair work B17 - Report on targets of completed work scheduled B18 - Daily work schedule drafting B19 - Elaboration of long-term/medium-term work plans B20 - Pruning trees that interfere with the network B21 - OSMOSE Maintenance B22 - Substation Maintenance B23 - Repair of hazardous situations B24 - Performing study on network improvements B25 - Performing Study on Agricultural Electrification B26 - Planning of Network Developments B27 - Performing Study for housing open Substations B28 - Updating Network Plans B29 - Inspection of blocks of flats B30 - Inspection of simple installation B31 - Monitor processing of SAB B32 - Materials and Spare Parts Monitoring B33 - Equipment Monitoring B34 - Line Monitoring B35 - Warehousing and Transportation B36 - Load Monitoring on Transmission Lines and Substations B37 - Monitor L/V Loads B38 - Monitoring Maintenance of Vehicles B39 - L/V, M/V Network Monitoring B40 - Preventive Maintenance of Network Elements B41 - Handling Electricity Stealing B42 - Handling fire near Distribution Network
Fig. 7. Overview of District micro-processes according to the functional organisation
316
V. Kavakli and P. Loucopoulos
As illustrated in Fig. 7 in the majority of cases District customers contact the company through the Customer Services Section. To fulfil the customer demand there is a need to involve the Technical Section of the District. The service requested by the customer will be delivered by the Technical Section after authorisation by the Customer Electrification Section. By studying the District micro-processes one can easily conclude that while many activities are performed within different functional divisions they are parts of the same business process. For example micro-process A1:‘Electricity Supply * Application Fulfilment for the L/V Customers’ and micro-process B3: ‘Performing Study of Electricity Supply through modification of the L/V Network’ are parts of a bigger process which deals with the supply of electricity to
District customers. However, this is not obvious in the functional organisation description since there is no description of the interrelationships between different functions. In order to understand the interactions between the functional unit described in Fig. 7 we proceeded to modelling the current District behaviour in terms of actor-role diagrams of District activities. An actor-role diagram presents a high-level view of the association between actors and their different roles. An example of an actor-role diagram for the A1:‘Electricity Supply Application Fulfilment for the L/V Customers’ is illustrated in Fig. 8. This diagram describes the actors involved in supplying electricity to a L/V customer. This is a core District activity and is realised through the co-operation of several District actors. This co-operation is modelled in terms of dependencies between roles. There are two parties involved in the dependency: the requester role, i.e. the one that needs something in order to fulfil its responsibilities, and the provider role, i.e. the one that can provide the missing component. This relation can be of various types: (a) authorisation dependency denotes hierarchical dependencies that can exist between roles; the provider role gives authorisation to the requester role, (b) goal dependency reflects the fact that the achievement of a goal that the role brings about is dependent on the achievement of a goal of another role, (c) coordination dependency expresses the need for one role to wait for completion of another role’s responsibilities before it can complete its own, and (d) resource dependency illustrates the need for one role to use a resource that can be provided by another role. role depends on the ‘service role for the achievement of its goal ‘to get connected to the electricity network’. On the other hand the ‘service administrative handling’ role depends on the ‘service requesting’ role for receiving money (which is a resource). Similarly the ‘service providing’ role depends on the
For example in Fig. 8, the
administrative handling’
*
L/V = Low Voltage
‘service requesting’
Goal-Driven Business Process Analysis - Application in Electricity Deregulation
317
role for authorising the construction of customer role depends on the ‘service administrative handling’ role for completing the contractual and financial tasks before it can give authorisation (co-ordination). ‘service
authorising’
installation. Finally, the
Technical Section
Customer Service Section
Customer
Service Requesting
Goal:
C
Administer servicing of customer’s request for electricity
G
R
Deal with contractual and financial matters
Actor
Legend
Role
Service Providing
Service Administrative Handling
Goal: To get connected to the electricity network
‘service authorising’
Goal: C
Investigate all technical parameters
A
Authorisation dependency
C
Co-ordination dependency
R
Resource dependency
G
Goal dependency
A
Calculate required materials and costs
Customer Electrification Section
Construct new customer installation and connect it to the network
Service Authorising Goal: Authorise service provision to the customer C
Deal with customer installation logistics
Fig. 8. Actor-role diagram concerning the ‘Electricity Supply Application Fulfilment for the L/V Customers’
The advantage of the actor-role diagram is that it provides a clear view of the interactions across different functional divisions. In this way it becomes apparent that fulfilling a L/V customer application for electricity supply is not solely the responsibility of the Customer Services Section but also depends on the co-operation of the Technical and Customer Electrification Section. Such interactions would appear as inputs/outputs in an activity-oriented view, thus obscuring the fact that ‘Electricity Supply Application Fulfilment for the L/V Customers’ cannot be performed independently of other activities performed by other sections. In addition the ability to include the customer role in the actor-role diagram, is a step
318
V. Kavakli and P. Loucopoulos
towards a process-centred view of the organisation in which each process has a customer. Service Requesting
Service Administrative Handling
Triggering Event: Submit application form
Service Providing Service Authorising
Send electrification service order to TS Inspect customer site
Perform electrification study Send study to CES for authorisation Consider offer
Inspect study
Contact customer
Calculate customer contribution Pay deposit
Receive payment
Sign contract Sign Contract Send customer details to CES Billing and Accounting Authorise construction Construct installation Install Meter Outcome: Notify customer: Application fulfilled
Customer
Customer Service Section
Connect Meter
Customer Electrification Section
Technical Section
Fig. 9. Role-activity diagram for the ‘Electricity Supply Application Fulfilment for the L/V Customers’
The actor-role diagram gives a first-cut view of the organisational aspects regarding the responsibilities of individuals or groups in their involvement in the operation of a business process. A more detailed view of these roles was constructed in terms of role-activity diagrams [Ould 1995]. These diagrams show the set of activities that are generally carried out by an individual or group within the context of their role. An example of a role-activity diagram for the ‘electricity supply application fulfilment for L/V customers’ is illustrated in Fig. 9. Role-activity modelling encourages the identification of the key operational components which can be measured (activity duration, actor skills, resource costing etc.). In that sense roleactivity models provide the context for performing evaluation of process efficiency.
Goal-Driven Business Process Analysis - Application in Electricity Deregulation
5.3
319
Construct Intentions from Roles
In section 5.2 we presented a small example of micro-processes organised according to the current functional structure of the electricity company. The key question however of “what are the types of processes and how should be logically organised?” still remains unanswered. Satisfy customer demand for electricity
Goal
Legend
Goal decomposition
Supply customers with electricity
Supply L/V customers with electricity
Deal with technical aspects
Deal with administrative issues
Administer servicing of customer request for electricity
Deal with contractual and financial matters
Deal with administrative issues
Authorise service provision to the customer
To investigate all technical parameters
Deal with customer installation logistics
To construct new customer installation and connect it to the network
To calculate required material and costs
Fig. 10. Partial view of District goals
We address this question by using the business goals that represent the intentions of processes. These goals are presented in the ‘body’ of each role in the actor role diagram in Fig. 8. Such goals are mainly operational goals, that is they are expressed in terms of business objects and activities that realise them. For example the ‘service providing’ goal to ‘construct customer installation and connect it to the network’ refers to the construct installation, install meter and connect meter activities identified in the role-activity diagram in Fig. 9. As explained in section 4.2 role goals represent low-level goals in a goal hierarchy. Having identified the role goals (i.e. those goals in the goal graph that represent operationalisations of the business objectives) there was a need to establish their causal relationships to higher-level goals. Starting from the goals identified in Fig. 8 we constructed the goal graph presented in Fig. 10.
320
V. Kavakli and P. Loucopoulos
District business goals Satisfy customer demand for electricity Satisfy load increase Reinforce/extend District network Indemnify field owners for damages caused to fields from the network Supply customer with electricity Supply L/V customer with electricity Supply M/V customer with electricity Alter characteristics of existing customer installation Stop supply of electricity to customers
Ensure product quality
Ensure safe and continuous network operation Restore network operation in case of failure Facilitate scheduling of repair works Isolate part of network for work execution Schedule shifts of failure repair team Execute repair works Ensure good network operation Avoid problems caused by contact of network with tree branches Gain clear picture of network condition Monitor materials and spare parts conveyance and performance Prevent electricity failures Ensure good operation of Substations Ensure good operation of network elements Monitor load and prevent overcharges Improve network quality
Ensure positive financial profit
Modernise agricultural exploitations Plan for network improvements Ensure realisation of plan targets Resite existing network that interferes with other works in the area
Ensure profit from services provided to customers Collect data on customer electricity consumption Verify correct functioning of metering devices Facilitate customer payment Collect customer payment Stop supply of electricity to non-paying customers Restart supply of electricity after customer’s debt is settled Ensure correct charging of customers Ensure correct calculation of electricity consumption Verify correct functioning of metering devices Respond to customer complaints concerning billing Verify and correct billing information Improve exploitation of company assets Prolong service life of wooden poles Efficient use of support equipment Improve network exploitation Improve vehicle exploitation Simplify the process of supervising and distributing materials Manage flow of materials and equipment Train District personnel Ensure safety of District Personnel Protect company’s interest Receive compensation for damages caused to distribution network from third parties Avoid stealing of electricity
Fig. 11. Overview of District business goals
In Fig. 10 it can be observed that goals related to the roles involved in fulfilling the application of a L/V customer for electricity supply, satisfy the higher goal ‘supply L/V customers with electricity’. This in turn supports the satisfaction of the goal ‘supply customers with electricity’ which ultimately supports the achievement of the strategic District goal ‘satisfy customer demand for electricity’.
Goal-Driven Business Process Analysis - Application in Electricity Deregulation
321
This process of abstracting from operational goals to higher intentions, naturally involved different stakeholders. The result was a clear, agreed view of the reasons for the business process (the WHY dimension, see also Fig. 5). By repeating this process for all District roles we delivered the following goal hierarchy that explains what the District is currently trying to achieve. A global view of the District goals and the associated activities is presented in Fig. 11. Each leaf goal in this goal tree refers to specific business micro-processes studied in the actorrole/role-activity modelling step. 5.4
Adopt a Process View, Driven by Discovered Goals
Using the goal graph illustrated in Fig. 11 District micro-processes are grouped in five core business processes each aiming to achieve a strategic business goal. These are: Table 1. List of District business processes Business Process Customer electrification Network reinforcement/extension Network operation Exploitation and maintenance of company assets Customer Billing
Business Goal Supply customer with electricity Satisfy load increase Ensure safe and continuous network operation Improve exploitation of company assets Ensure positive profit for services provided to customers
The business processes presented in Table 1 process correspond to the goals highlighted in Fig. 11. Each process is composed by the micro-processes that realise the subgoals of the goal that is realised by the entire process. Thus the map of District micro-processes is re-organised in terms of the goals these activities aim to achieve. The result is illustrated in Fig. 12. In contrast to the ‘vertical’ map illustrated in Fig. 7, Fig. 12 presents a ‘horizontal’ view of the District whereby each District process crosses functional boundaries and requires the collaboration of more than one District sections. Indeed it can be seen in Fig. 12 that for each business process there is a horizontal referencing, including micro-processes from two or more District sections (shown as A, B and C and D in Fig. 12). For example the ‘customer electrification’ process involves the collaboration of the Customer Services Section, the Technical Section and the Customer Electrification Section.
B32- Performing Study of Electricity Supply through Modification of L/V Network B4 - Performing Study of Electricity Supply without Modification of L/V Network C1 - Electricity Supply Application Fulfillment for M/V Customers B3 - Performing Study for New or Existing M/V Customers B30 - Inspection of simple installation B29 - Inspection of blocks of flats A4 - Meter Re-connection A6 - Installation Modification A7 - Failure Restoration A3 - Meter Disconnection Customer C3 - Installation Dismantlement Services
A Section
Network reinforcement/extension B9 - Network Modification B5 - Performing Study on Network Modification B10 - New 20/0.4 KV Substations Construction B11 - New U/G 20 KV Line (non Attica) Construction B12 - New Building Construction for 20/0.4 KV S/S B27 - Performing Study for housing open Substations B28 - Updating Network Plans C2 - Handling field damages
(Agencies)
Technical Section B
Customer
C Electrification Section
Customer billing A12 - Meter Reading A5 - Meter Check & Maintenance A8 - Grant Bank Payment Authorisation A9 - Revoke Bank Payment Authorisation A11 - Payment Collection B7 - Electricity Disconnection B8 - Electricity Re-connection A10 - Billing Correction
A7 - Failure Restoration B23 - Repair of hazardous situations B42 - Handling fire near Distribution Network B6 - Repair of Damages to Network due to Natural Disasters or Black out B16 - Drafting of monthly electricity failures repair work B14 - Drafting of provisions for a completed program B15 - Electricity failure repair by shifts B20 - Pruning trees that interfere with the network B31 - Monitor processing of SAB B34 - Line Monitoring B32 - Materials and Spare Parts Monitoring B13 - U/G Cable Re-routing for 20KV Lines B22 - Substation Maintenance B40 - Preventive Maintenance of Network Elements B36 - Load Monitoring on Transmission Lines and Substations B19 - Elaboration of long-term/medium-term work plans B18 - Daily work schedule drafting B17 - Report on targets of completed work scheduled B26 - Planning of Network Developments B24 - Performing study on network improvements B25 - Performing Study on Agricultural Electrification A2 - Network Resitation
D Personnel Management Section DISTRICT
Exploitation and maintenance of company assets B21 - OSMOSE Maintenance B33 - Equipment Monitoring B35 - Warehousing and Transportation B37 - Monitor L/V Loads B38 - Monitoring Maintenance of Vehicles B39 - L/V, M/V Network Monitoring D1 - Personnel Training D2 - Prevention of industrial accidents B1 - Handling Damages caused on PPC’s Networks by third parties B41 - Handling Electricity Stealing
Fig. 12. District process map based on District goals
6
Discussion
In this paper we have presented an approach to business process modelling based on the notion of ‘process abstraction through intentional affinities’. The purpose of business process modelling is ultimately the improvement of the enterprise in order to deal with change. In general therefore, we advocate that change management should be seen as the process of identifying business goals and relating business processes to these goals. Returning to the District application, capturing goals for change presented a number of problems stemming primarily from: (a) the uncertainty of the future situation and (b) the different perceptions that different District personnel had on the issues for change. Relating goals to existing processes proved to be the only reliable way to achieve a clear view of the issues for change. The result of this exercise was: (1) to increase the awareness of District personnel about the issues for change; (2) to give the opportunity to strategic and operational
Goal-Driven Business Process Analysis - Application in Electricity Deregulation
323
District personnel to participate in the creation of the company vision; and (3) to produce a list of strategic goals for change. Relating change goals to existing change processes helps to further refine goals based on the specific business process characteristics. For example ‘reduce response time of any customer request’ can be refined in terms of measurable goals that refer to specific micro-processes involved in the customer electrification process, e.g., ‘reduce period from customer application to District offer to 10 days’, ‘reduce time necessary to construct a new installation to 50 days’, and ‘reduce period from customer payment to meter connection to 40 days’. In addition, the fact that the customer electrification micro-processes have already been modelled in terms of role-activity diagrams, can further assist in evaluating the reasonableness of these proposals and also to reveal the points where process delay occurs and suggesting improvements. Of course in some cases it is not possible to identify any existing process that can be related to the goal for change. For example ‘build a commercial profile’ objective refers to processes that are completely new for a company that did not have to deal with competition up to now. In this case the new process should be defined from scratch again as a causal transformation of business goals (see Fig. 5). The goal of a process organisation is to create a high performance workplace, a high quality work environment noted for excellence in efficiency, effectiveness and customer satisfaction. With a focus on process, it is very common to see process organisations managing interdisciplinary work teams instead of specialised units seen in traditional organisation of enterprises. The approach presented in this paper recognises the need for a co-operative approach to reaching a common view of the products and services delivered by the business processes of an enterprise. In recent years similar approaches have been advocated in the areas of enterprise modelling [Jarke, Bubenko, et al 1993];[Yu 1994];[Kueng and Kawalek 1997] and CSCW [Ellis and Wainer 1994]. The focus of these approaches is on individual agents and the goals that drive and justify their individual activities as well as their cooperation. Whilst our approach shares this direction, we also advocate a more holistic framework whereby, business goals as well as individual actors’ goals are considered in terms of systematically analysing current business processes, the goals for change and the impact of these goals on existing or new business processes. Such a goal-driven, deliberative approach presents, in our opinion, a major step towards meeting this emerging challenge of managing business change.
7
Acknowledgements
The work reported in this paper has been partly supported by the commission of the European Union under the ESPRIT programme. The authors wish to acknowledge the assistance of Mr D. Beis and Mr G. Vgontzas, as well as the participation and
324
V. Kavakli and P. Loucopoulos
collaboration of Professor C. Rolland, Dr S. Nurcan and Dr G. Grosz, in the industrial application described in this paper.
References Alderman, N., Maffin, D., Thwaites, D., Vaughan, A.T., Braiden, P. and Hills, W. (1997) Understanding customer value: a business process analysis approach, MESELA'97, Loughborough, 1997. Bubenko, J. (1994) The F3 Reference Manual,, Deliverable F3, version 0.4, June, 1994. Davenport, T. (1993) The Process Innovation, Harvard University Press, Cambridge, MA, 1993. DeMarco, T. (1978) Structured Analysis and System Specification, Yourdon Inc., New York, 1978. Dobson, J.S., Blyth, A.J.C., Chudge, J. and Strens, R. (1994) The ORDIT Approach to Organisational Requirements, in 'Requirements Engineering: Social and Technical Issues', M. Jirotka and J. A. Goguen (ed.), Academic Press, London, pp. 87-106. Easterbrook, S. and Nuseibeh, B. (1995) Managing Inconsistencies in an Evolving Specification, RE'95, IEEE Computer Society Press, Los Alamitos, California, York, England, 1995, pp. 48-55. Ellis, C.A. and Wainer, J. (1994) Goal-Based Models of Collaboration, Collaborative Computing, Vol. 1, 1994, pp. 61-86. Franckson, M. and Peugeot, C. (1991) Specification of the Object and Process Modelling Language, , ESF Report D122-OPML-1.0, 1991. Hammer, M. and Champy, J. (1993) Reengineering the corporation - a manifesto for business revolution, 1993. IDEF0 (1993) Integration Definition for Function Modeling (IDEF0), Computer Systems Laboratory, National Institute of Standards and Technology, FIPS Pub 183, Dec 21, 1993, 1993. Jarke, M., Bubenko, J., Rolland, C., Sutcliffe, A. and Vassiliou, Y. (1993) Theories Underlying Requirements Engineering: An Overview of NATURE at Genesis, IEEE International Symposium on Requirements Engineering, IEEE Computer Society Press, San Diego, California, 1993, pp. 19-31. Kueng, P. and Kawalek, P. (1997) Goal-Based Business Process Models: Creation and Evaluation, Business Process Management Journal, Vol. 3, No. 1, 1997. Loucopoulos, P., Kavakli, V., Prekas, N., Rolland, C., Grosz, G. and Nurcan, S. (1997) Using the EKD Approach - The Modelling Component, UMIST, WP2/T2.1/UMIST/1, April 1997, 1997. Ould, M. (1995) Business Processes: Modelling and Analysis for Re-engineering and Improvement., John Wiley & Sons, Chichester, 1995. Ross, D.T. and Schoman, K.E. (1977) Structured Analysis for Requirement Definition, IEEE Transactions on Software Engineering, Vol. SE-3, No. 1, 1977, pp. 1-65. Singh, B. and Rein, G.L. (1992) Role Interaction Nets (RINs): A Process Definition Formalism, MCC, Technical Report, #CT-083-92, July 1992, 1992. Swenson, K.D. and Irwin, K. (1995) Workflow Technology : tradeoffs for Business Processes Re-engineering, Conference on Organisational Computing Systems COOCS 95, CA, 1995. Yu, E. (1994) Modelling Strategic Relationships for Process Reengineering, Ph.D., University of Toronto, 1994.
Real-time Information System for Risk Management on Motorways Tullio Tanzi(1,2), Sylvie Servigne(1), Régis Guiol(3) (1)
Laboratoire d'Ingénierie des Systèmes d'Information, INSA de Lyon, 20 Avenue Einstein, Villeurbanne Cedex F-69621 [email protected] (2) Etablissements Jean GRANIOU, ZI des Trois Moulins, BP 717 Antibes Cedex 1 F-06633 (3) Direction Recherche & Développement ESCOTA, BP41, Mandelieu Cedex F-06211
Abstract. Every day more and more people and goods have to be transported rapidly, safely and at a lower cost. Continuous economic growth brings a traffic volume increase and implies a gradual saturation of road and motorway networks. Managing motorways is today a complex task, especially during a time of crisis (traffic jam, accident). Our aim is to design a system for risk management on motorways. Risk management is a very important endeavour today and many people work on this subject [Adams 95], [Beek 92]. In our system, risk management provides tools to help managers to anticipate potential accidents so as to avoid them or, otherwise to reduce accident consequences. In addition, the system must be able to give fit information to initiate preventive actions and then to follow them in real-time. Such a system requires much information in real-time. Information must be acquired and then sent to the various motorway departments to be processed. It is necessary to have a complex real-time architecture based on sensors and communication technology to link motorways with the management stations. The proposed global system for risk management on motorways is presented in this paper. Real-time information composition and use particulars are presented. Details on processing can be found in [Tanzi 97b]. An industrial prototype has been realised and for the motorway network of the south of France.
1
Introduction
In managing toll motorways a complex chain of decisions has to be taken by operators when an accident occurs in order to rapidly restore to a normal traffic. Indeed, even a minor incident (the fall of luggage from a car, a small collision) can have consequences that hold up the traffic during several hours after the end of the incident [Boulmakoul 93]. The number of cars caught in the resulting traffic jam depends on the time needed to repair the carriageway. The delay between the incident and the arrival of intervention vehicles implies increasing difficulties in accessing to the incident location because of the traffic jam [Tanzi 97a]. As lanes are one-way lanes, it is very difficult to use lanes in the opposite direction, even for vehicles authorised to do so. To come back to a normal situation, it is then necessary to evacuate the cars caught in the traffic jam, which may last a long time. Various studies on traffic control show that B. Pernici and C. Thanos (Eds.): CAiSE’98, LNCS 1413, pp. 325-338, 1998. Springer-Verlag Berlin Heidelberg 1998
326
T. Tanzi, S. Servigne, and R. Guiol
a reduction of 30% of the total intervention duration implies a reduction of 55% of the lost time of the drivers [HCM 95]. To obtain a realistic overview of the situation on the motorway, motorway managers receive traffic data from measuring sites at their disposal. Measuring equipment are numerous and set along the motorway at frequent intervals. To make an optimal use of traffic data, it is necessary to send them rapidly from an acquisition station to the processing station and to assemble these data from several acquisition stations to produce a global overview of motorway areas. Improving the speed of data transmission allows operators to react rapidly to incidents. It is so vital to have a realtime system. Depending on the type of incident and existing information concerning it, it could be possible to anticipate the potential accident so as to avoid it by initiating preventive measures. Nevertheless, if it is not possible to avoid accident, preventive actions could still reduce or ameliorate consequences. Our aim is to build a system for risk management on motorways. A sine qua non condition for a system able to be successful in preventing motorway risk is to obtain immediately information which characterise traffic and environmental conditions. The aim of the paper is to present our proposed architecture for a real-time system. First, information characterising traffic will be described. Then the real-time system will be presented. After a general presentation of the system context, information flow and processing will be described, going from the acquisition process to their use in the decision phase. The prototype has been realised on the Escota's motorway network. The Escota company is in charge of a part of the motorways of the south-east of France.
2
Traffic Data
Traffic is characterised by a flow (distribution of vehicles in time), an business index of a given road segment (concentration) and the vehicle speed. Different sensors are used by traffic managers for traffic data acquisition. The most common procedure for traffic data supervision consists in using magnetic loops buried in the road surface, which detect the passage of metallic masses (vehicles). Two loops situated successively in the same lane (Figure 1) allow acquisition of vehicle speed.
Real-Time Information System for Risk Management on Motorways
Calculation Station
327
Adjustable Detection threshold
Voltage V d t2
t1
l
Time
t1 : The front of the vehicle goes along the magnetic loop : Front Rise
Two Ways Road Example
Speed is obtained by v = tl2+−dt1
t1-t2 : The vehicle passes over the loop : Constant amplitude t2 : The backside of the vehicle passes the loop : Front Fall
Fig. 1. Calculation station The processing of the data produced by various pieces of equipment that already exist on the motorway network gives an image of the real situation. The Figure 2 shows an example of traffic properties of a given motorway lane. Station 23 23 23 23 23 23 23 23
1 1 1 1 1 1 1 1
Weight 0 0 0 0 0 0 0 0
Time Date 20/04/97 00:30:00 20/04/97 00:36:00 20/04/97 00:42:00 20/04/97 00:48:00 20/04/97 00:54:00 20/04/97 01:00:00 20/04/97 01:06:00 20/04/97 01:12:00
Flow 800 800 720 740 700 590 620 630
1 1 1 1 1 1 1 1
Speed 115 112 113 112 108 114 111 108
1 1 1 1 1 1 1 1
Oc rate 1 1 1 2 1 1 1 1
1 1 1 1 1 1 1 1
Truck Flow 0 5 3 5 0 1 0 13
1 1 1 1 1 1 1 1
District 11 11 11 11 11 11 11 11
Fig. 2. Example of real traffic data of a lane The processing of such data into a display form allows the preparation of three curves (speed, flow, and concentration) as seen in Figure 3. In this example, a first accident occurred at 10h40 am, and a second one occurred one hour after. Nevertheless, an inspection of these curves suggests that the risk of the first accident can be detected as early as 10h00 a.m.
328
T. Tanzi, S. Servigne, and R. Guiol 400
Flow 320
Concentration 240
Speed 160
80
0 0 1
2
3
4
5
6
7
8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24
Time (Hours)
Accident #1
Accident #2
Flow (veh/6 mn) Speed Concentration
Fig. 3. Display of traffic data Our aim is to use the delay between the risk detection and the potential accident to avoid this accident by implementation of a preventive action. For example, the vehicle speed may be reduced by displaying messages on dynamic roadsigns along the motorway. The most important action is to alert the motorway so that they can anticipate the potential crisis situation. More details concerning risk rate estimation may be found in [Tanzi 97b] and [Oppe 90].
3
Real-time system for traffic control
More and more sophisticated equipment constituting a global information acquisition system are available on motorways. It is necessary to adapt data exploitation tools to this context. This can be done taking real-time data acquired by sensors of the traffic management system into account. Then data are used for various purposes at various levels of the hierarchical motorway organisation (Technical Office, District, Control Station) in order to produce a synthesis to help managers to make a decision. Between the acquisition of data and its use for a decision, it is necessary to implement a data processing sequence. 3.1 Information use The various informations acquired along the motorway may be used for different purposes and by different departments (Figure 4). First, information is used by departments in charge of the viability of the network in order to maintain the quality of level of services. Information is also used during a crisis situation when an incident occurs, to verify if the proposed solution is appropriate to the incident. This is possible using people knowledge about conditions about the accident site. Strategy definition for the management and intervention by operators is easier using these
Real-Time Information System for Risk Management on Motorways
329
information. After the crisis, the chronological account of computing events is used for training, enriching the management organisation experience using the details of the stored information. In future, these data could be analysed and taken into account into the development of long-range action plans so as to avoid accidents and to manage accident situation better.
Fig. 4. Use of acquired information To understand and simulate traffic variations, numerous parameters must be taken into account at the same time because of data and context variety, but also due to mechanism complexity. It was necessary to design a hierarchical method so as to enhance the impact of parameters on the global process. The resulting method allows information filtering. All non significant-information (low impact on the phenomenon) is rejected. The information kept depends on a threshold value which has to be determined according to a desired precision. 3.2 Risk management At the beginning, an accident is often slight and controllable by the appropriate person, at the right time and at the right place. Concerning a motorway incident, the fit persons are the staff in charge of motorway. Telecommunications and computers allow decisions at a distance by means of a virtual visit to the right place. But what about the right time ? In the risk domain, a little problem may generate a lot of trouble bearing in mind that time operates against humans. We define the right time as the closest time to the event occurence, and, if possible, the best fit time would be the time just before the event. When an accident analysis is done a posteriori, we can realise how important event anticipation is, that is to say, to be at the right place, at the right time with the appropriate intervention measures. We chose to work on event prevention. We define prevention as the technical measures already located on the site when the incident occurs. The aim of prevention is to prevent from common risk and to limit the accident and its consequences.
330
T. Tanzi, S. Servigne, and R. Guiol
3.3 Real-time decision process When an incident occurs, real-time information allows verification of how appropriate resources are for the problem. Real-time data simulation tools gives a projection of the incident evolution and so allows identification the gravity of the incident. Action plan implementation and strategy definition are also based on the real-time data. Figure 5 shows the event management in real-time. Recording information also facilitates building the incident memory and enriching the organisation experience. In the future, modifying generic intervention plans by this experimental data will improve an operators ability to confront an accident.
data acquisition
Event
Analysis
Actions
Optimal decision
Database
Long time memory
Fig. 5. Event management in real-time The decision process is composed of steps (Figure 6) which are not always sequential. Sometimes, backtracking is necessary. The first step is the relevant data acquisition. The aim is to collect as much information as possible concerning the problematical event that occurred. Next, the conception phase consists in developing one or more a priori possible solutions and scenarios. At this step, additional information can be required which implies a second acquisition process (backtracking). The third step consists in choosing the best solution among the list of alternatives previously built. The final step, the evaluation process, consists in evaluating the previous choices and decisions. The aim of the final step is to correct if necessary the entire reasoning.
Real-Time Information System for Risk Management on Motorways
331
Fig. 6. Decision process To validate the proposed architecture, a prototype has been made on Escota's motorway. The prototype is devoted to help the operator follow the evolution of conditions which can imply an accident occurrence and so to detect automatically geographic areas where the risk of accident is high. All information and calculated data are displayed in real-time and continuously. Data are issued from acquisition stations (Figure 7) and meteorological stations. Other information, for example concerning road works maintenance, is acquired through communicating with an existing information system.
High Speed Network
Central System
Data Acquisition Equipment
Operators displays Fig. 7. Data acquisition stations
4
Architecture of the real-time system
In analysing traffic data, the automated system may detect conditions which are going to favour the increase of the risk level of every motorway segment. When a threshold of the risk is passed, the system is setting into a state of pre-alert. Information (motorway synopsis, flow, vehicle speed, concentration (Figure 14)) are displayed automatically and continuously on a specific screen. The risk index is computed and displayed in real-time. The operator is able to follow in real-time the evolution of the conditions and so to decide what to do to prevent or reduce consequences of a potential incident. To realise these operations, a specific architecture is needed.
332
T. Tanzi, S. Servigne, and R. Guiol
rk
etwo
District 1
High
dN Spee
District n
LAN
LAN
Control center Emergency Phone Network
Fig. 8. System architecture along the motorway Figure 8 gives an example of a potential global system architecture for a motorway. This architecture requires use of information issued from traffic data instruments at various levels of the hierarchical motorway organisation (Technical Office, District, Control Station). Information flow and according processing offices are graphed in the Figure 9. The foundation consists of Acquisition Stations in charge of data acquisition. Then, Technical Offices have to concentrate these data. A local analysis is undertaken at the district level. Finally, data consolidation is made by the Control Station. Consolidation Local Analysis Concentration Acquisition
Control Center District 1 District n LT 1
Equip 1
...
LT n Equip n
Fig. 9. Information flow of motorway organisation
Real-Time Information System for Risk Management on Motorways
333
4.1 Data acquisition Data acquisition is realised in real-time by sensors (as previously described in the first part) and characterises each passing vehicle. Acquisition Stations are in charge of realtime acquisition and then data transmission to the Technical Offices upon which they depend. 4.2 Data concentration Concentration of acquired data is realised by various Technical Offices. Each station corresponds to a specific geographical area of the motorway. First, data are stored locally and sent every night to a the control station by batch processing. The chaining processing is described in Figure 10. Upon arrived at the Technical Office, data are analysed to produce a synthesis of every Acquisition Station situation. Meteorological data are integrated here into the analysis process. As previously said, processing depends on climatic conditions, for example, a safe distance between vehicles depends on road-grip characteristics. Meteorological data may be furnished by a local station or acquired from a distant meteorological data server. Traffic Sensors Real Time Acquisition
Meteorological Data
Acquisition
Meteorological Stations
Analyse Alert High-Level
Local Saving Synthesis Elaboration Individual Data
Real-Time Alert Low-Level
Transfer Data Transmission Network
1 minute Data Agregation Agregations 6 minutes Data Agregation
Local Storage
Batch Transfer
Fig. 10. Data acquisition : processing sequence Depending on the type of acquired data, a pre-alert (low-level) or alert (high level) situation may be generated for every acquisition station federated by a Technical Office. When a pre-alert or alert situation is detected, the data transmission process is realised. It consists in transferring data issued from every Acquisition Station to the corresponding district. Transferred data are raw data (acquired data) or aggregated data (for example, for a period of one or six minutes). The choice of the most adequate data is done according to the alert level of every Technical Office.
334
T. Tanzi, S. Servigne, and R. Guiol
Generally, data are transferred every six minutes during a pre-alert phase and every minute during an alert phase. It can be changed by intervention by a district operator. 4.3 Data local analysis The local analysis is made by Districts. A District is an administrative entity representing about 40 kilometres of motorway. Every District receives a data synthesis from the Technical Offices. A cyclical observation of the traffic is made for every station. The frequency of data scanning depends on the alert level of every station. Figure 11 presents the processing sequences for the traffic observation.
Basis Calculation
Observation station n
Traffic Observation YES YES
Evolution ?
NO OK ?
Change ?
YES
NO
Alarm station n
Pre-alarm station n NO
NO
Focus station n Operator Actions
YES Other alarm ?
YES
End alarm ?
NO
Fig. 11. Traffic observation processes The traffic observations allows in real-time verification of the evolution of the traffic conditions. The display given as figure 12 shows an interface of our prototype during supervision time. Meteorologic indicators Road work Normal traffic conditions
First threshold reached
Second threshold reached
Fig. 12. Supervision Time interface
Real-Time Information System for Risk Management on Motorways
335
When index values reach a defined threshold, the system moves into a pre-alert phase and automatically switches the information onto a specific screen. This operation is devoted to attract attention of employees. When a second threshold value (more important than the first one) is reached, an alert phase is initiated. Data acquisition and calculation (risk index) are real-time and continuous processes so information are always displayed. The value of the alert threshold has been defined by various surveys of motorway companies, and estimated at 90% of the maximal capacity of the traffic. To keep enough time for the pre-alert phase, it is important for the first threshold value to be lower than the alert threshold (usually between 10% to 15% less). Figure 13 shows our prototype interface for an alert phase. In addition to the various indexes already represented in figure 12, numerous windows that allow to following the data evolution in detail are displayed. Every window corresponds to one data acquisition station. When a incident occurs, managerial operators of the incident can see on the screen the impact and evolution of the event on the traffic.
Fig. 13. Alert phase interface 4.4 Data consolidation Data consolidation is done by the Control Station of the motorway. Information concerning road works on the motorway are taken into account by the Control Station. To do this, the system uses data issued from various external databases belonging to different departments of the motorway, like the road works planning department.
336
T. Tanzi, S. Servigne, and R. Guiol
Taking into account road-works information, the Control Station is able to balance synthesis elaborated by districts. The Control Station has a unique and global vision of the entire network of the motorway at it's disposal. Nevertheless, the Control Station can have a localised display of one or more districts. Analyse
Operator Station
events
Main Database
Maintenance X Windows Network events
Servers
Risk Server Meteorologic data
station
SQL
station station
station
for X Window processes
Fig. 14. Architecture of Exploitation Support System for motorways Figure 14 represents a classical architecture of an Exploitation Support System for motorways. A "risk server" is added to complete the existing system In this case, system interoperability was realised based on classical database management systems and XWindows capacities for interface conception. 4.5 Preventive actions The risk rate estimation of infrastructure allows the creation of a strategy in order to reduce incident consequences or, if possible, to prevent the event. It is important to detect specific positions where the traffic is quite saturated with high speed vehicles and too short distance between vehicles to avoid a "pile-up". When such a position is detected, the target of a preventive action is to make drivers reduce their vehicle speed. This can be done using electronic boards displaying dynamic messages along the motorway. It is also possible to prevent traffic congestion by lane stop using an automatic folding arm (BRA [Guiol 94] : "Biseau de Rabattement Automatique") as already exists on the Escota's motorways.
5
Conclusion
As more and more people utilise motorways, the risk of traffic jams and accidents increases. When an accident occurs, it is very difficult to obtain information and to
Real-Time Information System for Risk Management on Motorways
337
intervene rapidly. The aim of our system is to help motorway managers in real-time to prevent accidents. If it is too late for prevention, then it is valuable to help them to build their intervention strategy and to follow their implementation. Today more and more electronic equipment along motorways and the substantial data produced bring out communication difficulties. It is important to organise processing as parallel processing in order to have a real-time working situation. The aim of the architecture of the real-time information system presented was to optimise the communication network capacity. Only useful data (concerning raw data or data synthesis) are sent on the network. Only appropriate data are given to operators if a pre-alert or alert situation is detected. The data transfer depends on the motorway situation (normal : no transmission, pre-alert and alert : transmission) and the system moves automatically from one state to another according to received and calculated data. Data are permanently displayed and, for efficiency, are graphical. The real-time aspects of the system allow people to react rapidly and sometimes to anticipate a potential crisis. A risk index has been elaborated to detect traffic conditions (normal, pre-alert, alert), it was not presented in this paper. More details can be found in [Tanzi 97b]. The experimental prototype checked the feasibility and coherence of the proposed system (including risk index). To do this, we achieved by setting up simulations using fictitious data selected with the aid of the operations staff of the Escota Company. Then we set up a control set consisting of simulated actual situations. We were thus able to estimate the accuracy of the system's reaction. Now we are working on an integration of our system into an Exploitation Support System as it is used in the motorway domain.
References [Adams 95] Adams J., Risk, University College London Press, London 1995. [Beek 92] Beek U., Risk Society, Ed. Sage, London 1992. [Boulmakoul 93] Boulmakoul A., Selam S. Intelligent Intersection : Artificial Intelligence and computer vision techniques for automatic incident detection. Artificial Intelligence in Traffic Engineering, 1993, VSP International Sciences Publisher, Zeist, the Nerdherlands. [Cohen 90] Cohen S. Ingénierie du trafic routier, Presse de l'ENPC, 1990. [Guiol 94] Guiol R., Neutralisation automatique de voies rapides et autoroutes. Revue des Associations des Ingénieurs et Anciens Elèves de l'Ecole Nationale des Ponts et Chaussées, PCM LE PONT n°12, Décembre 1994, p. 25-27 [Oppe 90] Oppe S., Koostra M. J., A mathematical theory for related long term developments of road trafic and safety. Proceedings of the 11 th International Symposium on Transportation and Traffic Theory. New-York, 1990. p.89-99 [Tanzi 97a] Tanzi T., Servigne S., A Real-Time GIS for Accident Prevention on Toll Motorways. Proceedings of JEC'97, Joint European Conference an Exhibition on Geographical Information, Vienna, Austria, April 16-18, 1997. p. 42-50
338
T. Tanzi, S. Servigne, and R. Guiol
[Tanzi 97b] Tanzi T., Guiol R., Laurini R., Servigne S., Risk Rate Estimation for Motorway Management. Proceedings of TIEMEC'97, The International Emergency Management and Engineering Society, Copenhagen, Denmark, June 10-13, 1997. p. 125-134 [HCM 95] Transportation Reseach Board. Highway Capacit Manual, Special Report 209, US Transportation Research Board, Washington D.C.. 1995
Describing Business Processes with a Guided Use Case Approach Selmin Nurcan, Georges Grosz, Carine Souveyet CRI, Université de Paris I Panthéon-Sorbonne, 90 rue de Tolbiac, 75013 Paris, France email : {nurcan, grosz, souveyet}@univ-paris1.fr Phone : +33 (0) 1 40 77 46 34, Fax : +33 (0) 1 40 77 19 54 Abstract : Business Process (BP) improvement and alike require accurate descriptions of the BPs. We suggest to describe BPs as use case specifications. A use case specification comprises a description of the context of the BP, the interactions between the agents involved in the BP, the interactions of these agents with an automated system supporting the BP and attached system internal requirements. Constructing such specifications remains a difficult task. Our proposal is to use textual scenarios as inputs, describing fragments of the BP, and to guide, using a set of rules, their incremental production and integration in a use case specification also presented in a textual form. The paper presents the structure of a use case, the linguistic approach adopted for textual scenarios analysis and the guided process for constructing use case specifications from scenarios along with the guidelines and support rules grounding the process. The process is illustrated with a real case study borrowed to an Electricity Company. Keywords : Business Process Description, Use Case Specification, Textual Scenario Analysis.
1 Introduction A Business Process (BP) is defined by Hammer and Champy in [8] as a set of activities which produces (from one or several inputs) an output valuable for the customer. For the sake of improving or re-engineering or simply understanding BPs, Hammer and Champy consider essential to start to describe them as accurately as possible. A BP can be described at different levels, each level corresponding to different types of BP requirements. First, a BP can be described with a set of interactions between agents involved in the BP, we call these interactions «organizational interactions». Agents can be either internal or external (e.g. customer, supplier) to the organisation where the BP takes place. Such a description can be completed by describing how an Information System (IS) supports, or shall support if the IS does not exist, the BP through the description of what we call « system interactions ». In such a description, the IS is considered as an agent. The BP description can be further refined and completed by adding the requirements of the « system internal ». These levels of description are summarised in figure 1. Similarly to Jacobson [9], we believe essential that "A tight, seamless relationship is required between the process that develops the business model and the process that develops the information system". Therefore, we consider that the development of the three levels of description must be considered seamlessly. Finally, the modelling technique to be used for describing BPs shall be forceful in that it should be possible .
.
The work presented in this paper is partly supported by the European Community within the framework of the ESPRIT LTR project CREWS (Co-operative Requirements Engineering With Scenarios) n°21903. B. Pernici and C. Thanos (Eds.): CAiSE’98, LNCS 1413, pp. 339-362, 1998. Springer-Verlag Berlin Heidelberg 1998
340
S. Nurcan, G. Grosz, and C. Souveyet
both to show easy-to-understand surveys of the BP and to describe parts of the BP in details [10], this complies with the different levels of description we propose. 1. organisational interactions
2. system interactions Data Base
3. system internal
* list of customers * geographical information (maps, charts, etc.)
Fig. 1. Three levels for describing business processes
On the one hand, we propose to describe BPs as use cases. As mentioned by Jacobson [10], use cases are a simple, natural way to identify and describe BP. Indeed, use case models are interaction oriented models that focus on the communications between the agents of an organisation. They are therefore very well adapted to the two first levels of description presented in figure 1. Complementary, use case driven approaches [2, 3, 9, 10, 19] have proved useful for requirements elicitation and validation. A use case is a description of one or more end to end transactions involving the required system and its environment [17]. The basic idea is to specify use cases that cover all possible pathways through the system functions [3, 19]. On the other hand, it is not sensible to imagine that a description of a BP can be obtained in one shot because BP are complex and involve many agents. Therefore, our proposal advocates for an incremental process for the description of BP. The incremental process guides, using a set of rules, the description of fragments of the BP and their integration into a single use case specification that includes all levels presented in figure 1. Therefore, a use case specification describes the context of the BP, the structured description of all interactions between agents involved in the BP (including the IS supporting the BP) and requirements about the IS. Finally, we propose to use scenarios as a means for describing the fragments of BPs. In both the Software Engineering (SE) and the Human Computer Interaction (HCI) communities scenarios are widely used as 'engine of design' [1, 15]. In the HCI community, they are used to elaborate on usability issues [7, 16, 26] and to help the expression and understanding of the goals of the design [12, 14, 22]. In the SE community, scenarios serve mainly as a front end to object oriented design [3, 20, 23, 24, 27]. Scenarios can be expressed in various ways : text, animation, prototypes, etc. but textual ones are recommended by several authors [5, 10, 13, 16, 18]. Natural language provides a simple means for describing a problem [12]. However, the lack of guidelines to support the process of constructing use cases is certainly one of the major drawbacks of use case driven approaches for requirements engineering [25]. In this paper, we propose an approach to guide textual use case authoring for the description of business processes. Our approach is an extension of the work described in [21] where the use case specification process was limited to the study of
Describing Business Processes with a Guided Use Case Approach
341
system interactions. We enlarge the scope of a use case for the description of the interactions between the agents of an organisation participating to a BP and therefore to describe what we call a "rich" use case. The input of the guided process is a set of textual scenarios describing the BP. The output is a use case specification of the BP, including organisational interactions, system interactions and system internal requirements expressed in a structured and non-ambiguous natural language text. Indeed, part of our approach relies on natural language analysis. Between the two extreme of using too constraining clauses templates (e.g. [2]) and completely free mode of expression (that increases the risks of ambiguity, inconsistency and incompleteness and makes automatic interpretation difficult), we chose a middle one. We propose to combine the use of narrative prose to express scenarios with structured natural language for the construction of "rich" use case specifications. The remaining of the paper is organised as follows. The use case model is briefly presented in section 2 along with its semantics, its associated linguistic patterns structures and an overview of the guided process. In section 3, we illustrate the process and the use of the rules with a real case study of a business process called "Electricity Application Fulfilment" (EAF) borrowed to an Electricity Company (EC). Finally we draw some conclusions and identify future work in section 4.
2 Overview of the Approach Central to our approach are the structure of a use case, the linguistic approach and the rules which ground the process of use case authoring. These three elements are described in turn. More details and examples about this approach can be found in [21]. 2.1 The Structure of a Use Case Because of the complexity of the use case structure, our approach proposes to construct use case specification incrementally taking into account partial stories called scenarios. These scenarios are the inputs provided by the use case author according to the structure depicted in figure 2. Context • Name • Goal • Initiating agent • Initial state Initial scenario
• Path of actions • Final state
Normal scenarios
Exceptional scenarios • Occurrence conditions
• Path of actions • Final state • Path of actions • Final state • Path of actions • Final state
• Path of actions • Occurrence conditions • Final state • Path of actions • Occurrence conditions • Final state • Path of actions • Final state
Fig. 2. The structure of the textual inputs of the guided process
In this description, there is first a contextual information which role is to situate the business process in its organisational setting (what Jacobson calls « company’s environment » in [10]. The context comprises the name of the BP, the description of the initiating agent (of the BP) who has a goal in mind. For instance, in the case of
342
S. Nurcan, G. Grosz, and C. Souveyet
the EAF process, the initiating agent is the customer of the EC whose goal is to be connected to the EC network. There are also some conditions required for the process to begin. For instance, the interactions of the EAF process does not start if the following condition does not hold : "the customer is in the EC office with a completed application form". Such conditions describe the initial state of the use case and are expressed in terms of agent or resource states. A resource can be physical (e.g. an identification paper) or abstract (e.g. a customer record). After having provided the contextual information, the use case author provides all possible scenarios. We distinguish two types of scenarios: normal and exceptional. A normal scenario describes one (possibly conditional) course (path) of actions reaching a final state that fulfil the goal of the initiating agent. An exceptional scenario is also described as a course of actions. However, the final state of an exceptional scenario does not fulfil the goal of the initiating agent. Actions can be either « system interactions » if one of the involved agent is the system supporting the BP or « organisational interactions » if the system is not involved. An action may be an atomic action or a flow of actions. An atomic action materialises an interaction from an agent to another agent and requires some resources. The sentence "the customer requests the commercial employee for a connection to the network" is an example of atomic action from the customer agent to the commercial employee agent. The parameter of this action is a resource ("a connection to the network"). We consider two types of atomic actions: communication actions between two different agents and internal actions involving a single agent. The previous example of action is an illustration of a communication action whereas the sentence "a technical employee performs the connection to the network" is an example of internal action. We distinguish four types of communication actions: service request, service provision, information request and information provision. In a service request action from A to B, an agent A asks a service to an agent B. Complementary, in a service provision action from A to B, an agent A provides a service to an agent B. "The customer requests the commercial employee for a connection to the network" is an example of service request action which is satisfied by the service provision action "the customer is informed that the connection to the network is done". An information request is a communication action where an agent is asking for some information. "The commercial employee asks the customer to sign the contract" is an example of information request which expects the performance of the information provision action "the customer gives to the commercial employee the signed contract". All scenarios are incrementally integrated in a use case specification. The purpose of the use case is to describe how the initiating agent can interact with other agents to achieve his/her goal. The internal representation of a use case specification is a set of episodes (see figure 3). An episode comprises a flow of actions and the corresponding final states. There are several possible final states for an episode and therefore, the flow of actions can include several paths to reach each of the possible final states. A flow of actions is a complex action composed of other actions, it is similar to the flow of events as defined by Jacobson [10]. The composition is based on the sequence, concurrency, iteration and alternative constructors. "The customer requests for a connection to the network, then the commercial employee asks his
Describing Business Processes with a Guided Use Case Approach
343
identification papers and the location of the house to be connected" is an example of a flow of actions comprising a sequence of two atomic actions (request and ask). "If the meter exists, a technical employee performs the connection to the network" is an illustration of an alternative flow of actions. Flow conditions are necessary to integrate several courses of actions in one complex flow of actions. In the last example, the flow of actions integrates the description of what happens when the condition "if the meter exists" is true. We distinguish two types of episode : normal and exceptional. An episode is said normal when each of its possible final states ensures the fulfilment of the user’s goal else it is said exceptional. An exceptional episode describes a « non normal » course of actions reaching a final state which does not fulfil the goal of the initiating agent. In the EAF example, the normal episode corresponds to the flow of actions ending to the connection of the customer to the network. "If customer record is written-off" starts the description of a non normal course of actions described in an exceptional episode. An exceptional episode has an occurrence condition and includes a reference to an action of the normal episode in which the occurrence condition can become true. A use case specification comprises a single normal episode which is the integration of all normal scenarios, and a set of exceptional episodes that are associated to exceptional scenarios. A use case specification may be connected to system internal requirements as described in the third level of figure 1. System requirements other than the ones which are formalised in a use case specification itself may emerge in the course of the specification process. "EC must have the land register information" or "EC must maintain the list of written-off customers" are examples of such system internal requirements. As mentioned earlier, all scenarios are provided in a textual form. Furthermore, a use case specification is also expressed in textual form. Consequently, there is a relationship between the text structure and the use case structure. The text structure can be decomposed into more elementary structures which are either clause structures or sentence structures. For example, the text "the customer requests for a connection to the network, then the commercial employee asks his identification papers and the location of the house to be connected" is a sentence structure decomposed into two elementary clause structures corresponding to the clauses : "the customer requests for a connection to the network", and "the commercial employee asks his identification papers and the location of the house to be connected". Sentence and clause structures correspond to the surface structures of the textual specification. They have a meaning which is respectively provided by sentence and clause patterns. The deep structure of the use case specification is provided by the sentence and clause patterns. Sentence patterns provide the semantics of sentences expressing sequence, conditional flows, etc. Clause patterns give the semantics of actions such as service provision actions, information request actions, etc. Sentence and clause patterns which are case patterns in a case grammar are presented briefly in the following section.
344
S. Nurcan, G. Grosz, and C. Souveyet
2.2 The Linguistic Approach The approach of use case specification presented in [21] is based on Natural Language (NL) text. There is thus, a necessity for catching the semantics of text. In order to fill the gap between the informal representation and the formal model of use cases, a Case Grammar [6] is used. It focuses on the notion of action which is central in the use case model and permits to catch both the semantics of NL and the semantics of the use case model. Following Fillmore’s approach [6], the case grammar introduces a set of semantic cases such as agent, object, destination, etc. Semantic cases define the semantic roles that the different elements of an action clause play with respect to the main verb. Semantic patterns are defined to associate a semantic case to the different elements of clauses and sentences. The purpose of the semantic patterns is to define the semantics of the clauses and of the sentences which are respectively expressed in the atomic actions and the flows of actions of use case specifications. At the level of clauses, case patterns are clause semantic patterns associated to verbs. Action clause semantic patterns provide the semantics of the atomic actions of the use case model by associating a semantic role to the related agents and parameter objects. State clause semantic patterns provide the semantics of object states on which rely the initial and final states and the flow conditions of flow of actions. Clause semantic patterns are presented in the form N (V) [C], where N is the name of the pattern qualifying the intrinsic semantics of the verb V, and C is the list of cases to be associated to the elements of the analysed clause. To represent the semantics of a clause in a use case specification consists in instantiating a clause semantic pattern. Identifying the concepts of the use case model from natural language is a two stage process which requires first the semantic analysis of the text and second the mapping of the resulting semantic patterns onto the concepts of the use case model. These two stages are illustrated in section 5 which describes more extensively the process of constructing the use case specification of the EAF example. Details and examples about the approach we use for natural language analysis can be found in [21]. 2.3 Guiding the Use Case Specification of Business Processes The process of use case specification of business processes is a stepwise process which guides the progressive transformation of input prose texts (starting with the initial scenario) into refined and structured texts and their integration in the use case specification. It comprises four main steps to : 1. define the context of the use case, 2. describe and complete the initial scenario, 3. integrate the scenario into the use case specification, and 4. prompt the need for new scenarios and guide their description and integration. During step 1, the use case is situated in its context by defining its name (the business process it describes), the initiating agent, the goal of the initiating agent and the initial state. Step 2 is an iterative one. It starts with the capture of the initial scenario. It is a text describing a course of actions that can be incomplete and ambiguous. It proceeds
Describing Business Processes with a Guided Use Case Approach
345
with the check and possible interactive completion of the initial scenario. The result of step 2 is a complete description of a pathway in an episode expressed unambiguously according to our semantic patterns. It corresponds to one path in the graph of episodes of the use case. During step 3, the completed scenario is integrated into the episode structure of the use case. Positioning the scenario in the use case specification is inferred from the study of the flow conditions. Performing step 2 and 3 may prompt the need for new scenarios (step 4) leading to iterate sequences of steps 2, 3 and 4. At this stage, a shift in the levels presented in section 1 (see figure 1) can be performed. For instance, as it will be shown in the next section while presenting the example, it is possible to shift from the description of "organisational interactions" to the description of "system interactions". The same applies to "system internal" requirements. The shift could also be the other way around (i.e. from a "system interactions" to "organisational interactions"). Guidelines and rules are defined to support the performance of each step. Guidelines help the author to perform the requested action. Rules define how to map textual scenarios onto the internal use case structure and to reason about it. The overall architecture of the process is sketched in figure 3. The front end is a guided communication with the user. The back end is based on rules working on the use case structure.
Internal Representation of the Use Case Specification
Fig. 3. Overall architecture of the process.
Guidelines have the form of plain text which can be prompted to the use case author on demand while writing down a scenario. They provide recommendation on the style of writing narrative prose. They suggest, for example, to avoid the use of anaphoric references, of synonyms and homonyms, ways for expressing an action, a conditional action, etc. They also advise the author on the expected contents of his/her prose. For instance, a guideline states that "The expected scenario prose is a description of a single course of actions. Alternative scenarios or exceptional treatments are described separately", etc.. There exists guidelines specific to levels 1 and 2 mentioned in section 1 (see figure1). For example, all interactions described at level 2 involve the computerised system supporting the business process as one of the agents. Rules are of five types : (1) analysis rules analyse the semantic contents of each sentence against the linguistic patterns; (2) mapping rules map the elements of
346
S. Nurcan, G. Grosz, and C. Souveyet
instantiated patterns into elements of the use case model, (3) refinement rules include the clarification and completion (3) integration rules help in positioning the scenario in the use case specification and (4) emergence rules prompt the need for other scenarios. Below, we exemplify the different types of rules mentioned above. More will be introduced in the next section with a walk through the EAF example guided process. All rules have a premise and a body separated by a «→ ». They are described by a first order logical formula. The premise defines the precondition for executing the body. This precondition is expressed on elements of the use case specification and defines when the rule is applicable. The body is either an action required from the author (using the ASK predicate) or the generation of new elements of the use case specification (using the GENERATE predicate). In the prototype tool under development, rules are implemented as PROLOG clauses. The enactment mechanism of the guided process is therefore, built on top of the PROLOG inference engine. More details about rules can be found in [21].
3 Guiding the Description of the Electricity Application Fulfilment Business Process This section describes the guided process of use case specification with the EAF (Electricity Application Fulfilment) example. The presentation is organised in five steps which show the progressive transformation, completion and integration of input scenarios describing the process of applying for electricity into a use case specification. With regard to the levels presented in section 1, in this example, we start with scenarios describing « organisational interactions » and end with the description of « system interactions ». Due to space limitation, the « system internal » level is not tackled in this paper. Let us assume that the use case has been situated and its contextual information provided and stored in the use case structure (see in appendix the corresponding specification). The first step that we consider is therefore, the capture of the initial textual scenario which is intended to describe a normal course of actions. 3.1 The Initial Scenario Capture Let us assume that after having studied the guidelines, the use case author writes down his view of the normal course of interactions between a user who applies for electricity and the Electricity Company (EC) as follows: The customer requests for a connection to the network. The commercial employee asks his identification papers and the location of the house to be connected. If the customer is not written-off and the installation exists the commercial employee asks to the customer to sign the contract and to pay the deposit. If the meter exists, a technical employee performs the connection to the network, but before the commercial employee sends a service order for connection. Then, the customer is informed that the connection to the network is done. The final states are : The customer is connected to the network. EC has a contract for this customer. EC has the network extended with a meter connection.
Describing Business Processes with a Guided Use Case Approach
347
The text is then scanned with the analysis rules and mapped onto elements of the use case specification using mapping rules. 3.2 Linguistic Analysis and Mapping on the Episode Structure As part of the linguistic approach sketched in section 2, analysis rules are used to identify the text structure, and to map the text onto instances of sentence and clause semantic patterns. For example the analysis rule AN1 aims at generating the action pattern from a clause having a subject and a verb expressed in the active form. AN1 : ∀ NG, V : [[NG](Subject)Object [V] (Main Verb)Action] (VG active)Action → GENERATE(Action(V)[Agent:? ; Object:NG])
Applying all necessary rules results in our example in the following set of instantiated patterns : Communication (request) [Agent:‘the customer’ ; Object:‘connection to the network’ ; Source:‘the customer’ ; Destination: ?] Communication (ask) [Agent:‘the commercial employee’ ; Object:‘his identification papers and the location of the house to be connected’ ; Source:‘the commercial employee’ ; Destination: ?] Constraint [ Condition : [State [Object:‘the customer’ ; State:‘not written-off’], State[Object:‘installation’ ; State:‘exists’]] ; Constrained : Sequence [Before : Communication (ask) [Agent: ‘commercial employee’ ; Object:‘to sign the contract’, Source: ‘commercial employee’; Destination: ‘the customer’] After : Communication (ask) [Agent: ‘commercial employee’ ; Object:‘ to pay the deposit’ ; Source: ‘commercial employee’; Destination: ‘the customer’]]]; Sequence[Before : Constraint [ Condition : State [Object:‘the meter’ ; State:‘exists’]; Constrained : Sequence [Before : Communication (send) [Agent: ‘commercial employee’; Object:‘a service order for connection’ ; Source: ‘commercial employee’ ; Destination: ?] ; After : Action (perform) [Agent:‘a technical employee’ ; Object:‘the connection to the network’]]] After : Communication (inform) [Agent: ?; Object: ‘the connection to the network is done’ ; Source : ?; Destination: ‘the customer’]] The final states are : State [Object:‘the customer’ ; State:‘connected to the network’] Ownership [Owner:‘EC IS’ ; Owned:‘contract for this customer’] Ownership [Owner:‘EC IS’ ; Owned:‘network extended with a meter connection’] Fig. 4. Instantiated patterns
The analysis of the initial scenario leads to the instantiation of seven action clause patterns within which six are communication action clause patterns. The instantiation of the action and communication action clause patterns from the input text provides values to the agent, source, object and destination cases. Question marks characterise missing elements in the input text ; they will be used for completion. The analysis rules identify the agents of the scenario out of the agent, source and destination cases : ‘the customer’, ‘the commercial employee’ and ‘the technical employee’. Then, the object case identifies the resources used in each atomic action of the flow of actions : ‘connection to the network’, ‘his identification papers and the location of the house to be connected’, ‘to sign the contract
348
S. Nurcan, G. Grosz, and C. Souveyet
and to pay the deposit’, ‘a service order for connection’, and ‘the connection to the network is done’. Moreover, the analysis of the initial scenario shows that the actions are related through two sequence sentence patterns and two constraint sentence patterns. Based on these pattern instances, mapping rules are automatically used to produce a natural language specification of the flow of actions of the initial scenario (consequently, it is a single course of actions). For sake of clarity, we use an indented presentation of the flow of actions, and we associate a unique identification number to each action. 1. The customer requests for a connection to the network. 2. The commercial employee asks his identification papers and the location of the house to be connected. 3. If the customer is not written-off and the installation exists 4. Then 5. The commercial employee asks to the customer to sign the contract 6. The commercial employee asks to the customer to pay the deposit. 7. If the meter exists 8. Then 9. The commercial employee sends a service order for connection. 10. A technical employee performs the connection to the network. 11. The customer is informed that the connection to the network is done Final states : The customer is connected to the network. EC has a contact for this customer. EC has the network extended with a meter connection. Fig. 5. Flow of actions and final states after the mapping of the initial scenario
Let us comment this mapping. First, based on the action and communication action clause patterns instantiated during the initial text analysis, atomic actions are identified through rules MA1, MA2 and MA3. MA1 :∀ V, A, O, S, D : Communication (V) [Agent:A ; Object:O ; Source:S ; Destination:D] ∧ (Unify (A, S) ∨ Unify (A, D)) → GENERATE(Atomic Action (Name : V, From Agent : S, To Agent : D, Parameter : O)) MA2 : ∀ V, A, ∃ O : Action (V) [Agent:A ; Object:O] ∧ Agent (O) → GENERATE(Atomic Action (Name : V, From Agent : A, To Agent : O)) MA3 : ∀ V, A, ¬∃ O : Action (V) [Agent:A ; Object:O] ∧ Agent (O) → GENERATE(Atomic Action (Name : V, From Agent : A, To Agent : A, Parameter : O))
As stated in these rules, the atomic actions are analysed separately, even if they occur in the same sentence. Communication action pattern instances lead to the mapping of an atomic action. Our purpose being not to rephrase the use case author scenario, the expression of the atomic actions identified in the initial text is not modified. Based on the sequence patterns, atomic actions have been organised in the right sequence. For example, the sentence "a technical employee performs the connection to the network, but before the commercial employee sends a service order for connection" has been split into "(9) The commercial employee sends a service order for connection. (10) A technical employee performs the connection to the network". When no explicit sequencing of the actions is expressed, the ordering of the sentences in the initial scenario is respected.
Describing Business Processes with a Guided Use Case Approach
349
Flow conditions such as "if the customer is not written-off" or "if the meter exists", are identified from constraint patterns. Once identified, the alternative flows of actions are isolated, and the corresponding flow conditions are put in the order provided by the constraint pattern instances. For example the sentence "If the meter exists, a technical employee performs the connection to the network, but before the commercial employee sends a service order for connection" becomes "(7) If the meter exists (8) Then (9) The commercial employee sends a service order for connection (10) A technical employee performs the connection to the network" . 3.3 Clarification and Completion of Actions The linguistic analysis provides a baseline for linguistic based clarification and completion of the identified atomic actions. Both rules rely on situations identifying possible linguistic ambiguities in the expression of the atomic actions. The clarification rules are used to change the wording and to remove possible ambiguities. Even if the first guideline recommends to "avoid the use of anaphoric references such as he, she, it, his and him", it is necessary to check systematically the text provided by the author. The grammatical analysis performed as a prerequisite for analysis rules provides the information required for these checks. Clarification rule CL1 uses this information and proposes to replace anaphoric references by nouns. CL1 : ∀ A : (Action [Agent:A ; Object:_ ] ∨ Action [Agent:_ ; Object:A ] ∨ Communication [Agent:_ ; Object:_ ; Source:A ; Destination:_ ] ∨ Communication [Agent: _ ;Object: _ ; Source: _ ; Destination:A] ∨ State [Object: _ ; State:A] ∨ Ownership [Owner:A ; Owned: _ ] ∨ Ownership [Owner: _ ; Owned:A] ∨ Localisation [Location:A] ) ∧ Anaphoric Reference (A) → ASK(« Clarify A by replacing this anaphoric reference by a noun ») Note : « _ » is used to denote an anonymous variable which value is of no importance. The predicate ‘Anaphoric Reference (A)’ identifies if the term A includes a pronoun (he, his, him, etc.).
The use case author has been using the pronoun "his" in the second sentence of his scenario. The clarification rule suggests to "clarify his by replacing this anaphoric reference by a noun". Taking this suggestion into account, he/she now decides to modify the flow of action 2 and replace "his" by "the customer’s" which is a more explicit resource name. The action (2) thus becomes "The commercial employee asks the customer’s identification papers and the location of the house to be connected". As explained in the previous section, the instantiated patterns may highlight missing parameters through question marks. Some of the completion rules (e.g. CO5) help avoiding this form of incompleteness. In the EAF example, several communication action pattern instances are in the situation of one or several missing elements. For example, as shown in the analysis section, analysing the atomic action "the customer is informed that the connection to the network is done" instantiates the pattern Communication (inform) [Agent: ?; Object: ‘the connection to the network is done’ ; Source : ?; Destination: ‘the customer’] where the agent of the action and the source of the communicated information are missing.
350
S. Nurcan, G. Grosz, and C. Souveyet
CO5 : ∀ V , ∃ O: Communication (V) [Agent : ? ; Object : O, Source : ? ; Destination : ?] / Atomic Action (V,_) → ASK(« Complete : V by ... (agent of the communication) from... (source of the communication) to... (destination of the communication) »)
Applying the completion rule CO5 displays the following template and asks the use case author to fill it in : "the customer is informed that the connection to the network is done by... (agent initiating the communication) from... (source of the communication) ". Making use of the template, the use case author completes the sentence which becomes "the customer is informed by the commercial employee that the connection to the network is done" in which the commercial employee is the agent and the source. The systematic application of linguistic completion rules leads to the completion of four actions. The new version of the current flow of actions specification is shown in figure 6 where the supplementary elements are in bold, and the elements that have been clarified in bold and italic. 1. The customer requests the commercial employee for a connection to the network. 2. The commercial employee asks to the customer the customer’s identification papers and the location of the house to be connected. 3. If the customer is not written-off and the installation exists 4. Then 5. The commercial employee asks to the customer to sign the contract. 6. The commercial employee asks to the customer to pay the deposit. 7. If the meter exists 8. Then 9. The commercial employee sends to the technical employee a service order for connection. 10. A technical employee performs the connection to the network. 11. The customer is informed by the commercial employee that the connection to the network is done. Final states : The customer is connected to the network. EC has a contact for this customer. EC has the network extended with a meter connection. Fig. 6. Flow of actions after linguistic clarification and completion
3.4 Completing Action Dependencies In the use case model, atomic actions are refined by several sub-types: service request, service provision, information provision, information request, internal action, etc. Based on this typology, we defined action dependency patterns which state dependencies between several types of atomic actions. The non respect of these dependencies is captured in the situations of the completion rules. Rule CO8, for example, states that the provision of a service S from an agent B to an agent A should be preceded in the flow of actions by the request of the service S from A to B.
Describing Business Processes with a Guided Use Case Approach
351
CO8 : ∀ V1, S, A, B : (Atomic Action(Name : V1, From Agent : B, To Agent : A, Parameter : S) ∧ (Type(V1) = ‘Service Provision’)) ∧ ¬(∃ V2 : Atomic Action (Name : V2, From Agent : A, To Agent : B, Parameter : S) ∧ (Type(V2) = ‘Service Request’) ∧ Follow(V1, V2)) → ASK(« Complete service provision V1 with the anterior action :... (service request S from A to B) »)
Similarly, any service request should be followed by a service provision, this also applies to information requests and provisions. These patterns are exploited in other completion rules which are similar to CO8. Standalone requests or provisions of services and information can thus be identified and completed if necessary with their counterpart. Another action dependency pattern establishes the dependency between an alternative action and the action of verification of the corresponding flow condition. The corresponding rule is triggered when this dependency pattern is not respected in the flow of actions allowing to associate a verification action to the condition. A systematic application of the associated rules on our flow of actions of figure 7 leads to the following specification, in which the new elements are emphasised in bold. 1. The customer requests the commercial employee for a connection to the network. 2. The commercial employee asks to the customer the customer’s identification papers and the location of the house to be connected. 3. The customer gives to the commercial employee the customer’s identification papers and the location of the house to be connected. 4. The commercial employee checks if the customer record is not written-off and the customer record exists and the installation exists. 5. If the customer record is not written-off and the customer record exists and the installation exists 6. Then 7. The commercial employee asks to the customer to sign the contract. 8. The commercial employee asks to the customer to pay the deposit. 9. The customer gives to the commercial employee the signed contract 10. The customer gives to the commercial employee the money for deposit. 11. The commercial employee gives back to the customer the customer’s identification papers. 12. The commercial employee checks if the meter exists. 13. If the meter exists 14. Then 15. The commercial employee sends to the technical employee a service order for connection. 16. A technical employee performs the connection to the network. 17. The technical employee informs the commercial employee that the connection is done. 18. The commercial employee sends to the customer a copy of the contract. 19. The customer is informed by the commercial employee that the connection to the network is done.
352
S. Nurcan, G. Grosz, and C. Souveyet
Final states : The customer is connected to the network. The customer has the customer’s identification papers. The customer has a copy of the contract. EC has a contact for this customer. EC has the network extended with a meter connection. Fig. 7. Flow of actions after the action dependencies completion.
Let us comment the incremental changes occurred between figures 6 and 7. As completion rules are based on types of actions, the use case author is first asked to provide atomic action typing. He/she thus identifies that : • the "request connection to the network" action is a service request *, • the "ask the customer’s identification papers and the location of the house to be connected" action is an information request, • the "ask to sign the contract " action is an information request, • the "ask to pay the deposit" action is a service request, • the "send a service order for connection" action is a service request, • the "perform the connection to the network" is an internal action, • the "inform that the connection to the network is done" action is a service provision *. The "ask the customer’s identification papers and the location of the house to be connected" information request action, does not have a corresponding information provision. At this point, the use case author thinks about describing the interactions between the customer and the actors of the EAF in a more detailed way. Thus, he/she inserts the following sentence in the flow of actions "The customer gives his identification papers and the location of the house to be connected". Using the linguistic analysis and mapping rules, the sentence is then converted into the atomic action 3 of figure 7. Indeed, the linguistic completion and clarification are also performed, as presented in previous section. This has led to replace "his" by "the customer’s", and to complete with a destination : "to the commercial employee". In the same way, the "ask to sign the contract" information request action is completed by the use case author inserting the following sentence "The customer gives to the commercial employee the signed contract". The same applies to "ask to pay the deposit". Finally, the "send a service order for connection" service request action from the commercial employee to the technical employee is completed by the corresponding service provision action "The technical employee informs the commercial employee that the connection is done". The application of the completion rules leads also to acknowledge that some parts of the specification are complete with respect to action dependency patterns. For example, the "request connection to the network" action is a service request from the customer to the commercial employee with the corresponding provision of the requested service (connection to the network). This correspondence is described by the use case author (see the two * in the list of typed actions provided above) at the same time than the types of actions. Thus, the description is already complete with respect to the satisfaction of a requested service. Note also that the use case author may decide not to apply the suggested completion. There are two flow conditions in the specification : (a) "the customer is not writtenoff and the installation exists", (b) "the meter exists". They do not have a preceding
Describing Business Processes with a Guided Use Case Approach
353
action for checking the flow condition. Asking the use case author to verify the need for two new actions for the condition checking leads to complete the episode specification with "(4) The commercial employee checks if the customer is not written-off and the installation exists" and (12) "The commercial employee checks if the meter exists". Using the corresponding completion rule, the use case author is asked to verify that the if these two conditions are complete. This leads him to insert another condition in the fourth sentence giving thus "(4) The commercial employee checks if the customer is not writtenoff and the customer exists and the installation exists". Final states have also to be completed, using the associated rule. The use case author is asked to verify their completeness and if needed provides the missing elements. With regard to the added elements, the use case author is asked to provide the necessary action for the final states to be reached. Similarly, the use case author is asked to verify that the conditions referring to agent names are either dealing with the agent itself or an abstract representation of the agent. For example, in action n° 4 of figure 7, the condition "the customer exists" may mean that either the customer is in the EC office or that he is known in the record of the company. This leads to replace "customer" by "customer record" giving thus" (4) The commercial employee checks if the customer record is not written-off and the customer record exists and the installation exists". 3.5 Reasoning on Flow Conditions A flow of actions issued from a scenario description involves several flow conditions. For instance, in the current version of the normal episode of the EAF use case, there are four flow conditions : - if the customer record is not written-off - if the customer record exists - if the installation exists - if the meter exists As a consequence of the scenario definition, this flow of actions together with the flow conditions constitute a pathway that permits the user to reach successfully one of the possible final states of the episode. In the case above, the flow of actions leads to an happy EC customer with the connection to the network. As presented in section 2, the normal episode may comprise several pathways, and be complemented by exceptional episodes. Each of them permits to reach one of the possible final states. We believe that reasoning on the flow conditions of the initial scenario description can help to discover exceptional episodes, and new pathways of the normal episode. The emergence rules based on flow conditions support this reasoning. These rules raise the questions of what happens if the flow conditions do not hold. Based on the flow conditions of a given pathway all combinations of negative conditions are calculated. Thanks to emergence rules, the use case author is first asked to characterise all of them being either normal, exceptional or meaningless. Second, if two combination of conditions lead to describe the same scenario, then it should be clarified by the use case author. The result of applying these rules leads to the following table.
354
S. Nurcan, G. Grosz, and C. Souveyet
(1) customer is written-off (2) (customer is not written-off) & (customer does not exist) & (installation exists) & (meter exists) (3) (customer is not written-off) & (customer exists) & (installation exists) & (meter does not exist) (4) (customer is not written-off) & (customer does not exist) & (installation exists) & (meter does not exist) (5) (customer is not written-off) & (customer exists) & (installation does not exist) & (meter does not exist) (6) (customer is not written-off) & (customer does not exist) & (installation does not exist) & (meter does not exist) (7) (customer is not written-off) & (customer exists) & (installation does not exist) & (meter exists) (8) (customer is not written-off) & (customer does not exist) & (installation does not exist) & (meter exists)
: exceptional : normal : normal : normal : normal : normal : meaningless : meaningless
For each normal and exceptional scenario, a new iteration in the specification process starts. This includes the activities 2, 3 and 4 presented in section 2.3 : the textual scenario description provided by the use case author, its analysis and completion and its integration in the use case specification. There are two kinds of integration : the integration of an exceptional episode, and the integration of a new course of actions in the normal episode. The integration of an exceptional episode is simple and consists in relating the exception to an action of the normal episode. The integration of a new normal course of actions in the normal episode requires to embed the new flow of actions in this episode. The guidelines proposed to the use case author during the capture of these new scenarios are the same as the ones used to capture the initial scenario. In addition, we offer a copy & paste facility which enables to duplicate in the scenario under writing, a course of actions which already exists in the specification. This facility is also used by the tool to provide to the author a partially written scenario description and ask him to complete it. The following text illustrates this step of the process for case (1). The use case author’s writing is in bold whereas the text automatically generated by the guidance tool is in italics, the use of the copy and paste functionality is mentioned as a comment between the signs " /* " and " */ ". /*copy & paste action(s) 1 to 2 of the normal episode */ The customer requests the commercial employee for a connection to the network. The commercial employee asks to the customer the customer’s identification papers and the location of the house to be connected. The customer gives to the commercial employee the customer’s identification papers and the location of the house to be connected. The commercial employee checks if the customer record is not written-off and the customer record exists and the installation exists. If ( the customer record is not written-off) Then the commercial employee informs the customer that he is written-off and the connection to the network can not be done. Final states : The customer is not connected to the network.
Describing Business Processes with a Guided Use Case Approach
355
The NL text is analysed and completed in a similar way as the one illustrated for the initial scenario. The resulting exceptional episode specification is the following. Exceptional episode of the use case Electricity Application Fulfilment Name : WrittenoffCustomer Occurrence condition : When (the customer record is not written-off). Where : the action 5 of the NormalCase episode. Action : 1. The commercial employee informs the customer that the customer record is written-off and the connection to the network can not be done. 2. The commercial employee gives back to the customer the customer’s identification papers. Final states : The customer is not connected to the network. The customer has the customer’s identification papers.
Proceeding in the same way with the scenario number 2 leads the use case author to describe the flow of actions when the customer does not exist, the installation exists and the meter exists as shown in figure 8. /*copy & paste action(s) 1 to 4 of the normal episode */ The customer requests the commercial employee for a connection to the network. ... If the customer record is not written-off Then If (the customer record exists) Then The commercial employee creates the customer record /*copy & paste actions 7 to 10 of the normal episode */ If (the installation exists) Then The commercial employee asks to the customer to sign the contract The commercial employee asks to the customer to pay the deposit. The customer gives to the commercial employee the signed contract. The customer gives to the commercial employee the money for deposit The commercial employee gives back to the customer the customer’s identification papers. The commercial employee checks if the meter exists. If (the meter exists) Then /*copy & paste actions 13 to 17 of the normal episode */ The commercial employee sends to the technical employee a service order for connection. A technical employee performs the connection to the network. The technical employee informs the commercial employee that the meter connection is done. The commercial employee sends to the customer a copy of the contract. The customer is informed by the commercial employee that the connection to the network is done. Final states : The customer is connected to the network. The customer has the customer’s identification papers. The customer has a copy of the contract. EC has a new customer. EC has a contact for this customer. EC has the network extended with a meter connection. Fig. 8.The scenario number 2 as provided by the use case author
356
S. Nurcan, G. Grosz, and C. Souveyet
Note that the copy & paste functionality is used in this case by the use case author to introduce actions 7, 8, 9, 10, 11, 12, 15, 16, 17, 18 and 19 of the normal episode in the current scenario. The analysis and completion steps are performed to obtain a specification that can be integrated in the use case specification. The integration step stands in two parts : the integration of the final states and the integration of the actions. The integration of the final states leads to add the keyword "sometimes" when a final state exists in the normal episode but not in the new pathway and vice versa. This results in adding "sometimes" before the final state "EC has the network extended with a meter installation" in the normal episode during the integration of scenario number 5. Then the integration rules are applied to re-organise the flow of actions of the normal episode. The integration of all the normal flows of actions, namely cases (2), (3), (4), (5) and (6) leads to the normal episode of figure 9. An asterisk marks a flow condition related to an exceptional episode. The same reasoning can be recursively applied to the new flow conditions (33) and (39). This leads to the emergence and specification of two exceptional episodes, namely "NetworkConnectionAborted" and "InstallationOnly" that are described in the appendix. The appendix presents all exceptional episodes of the EAF use case. Normal episode name : NormalCase action : 1. The customer requests the commercial employee for a connection to the network. 2. The commercial employee asks to the customer the customer’s identification papers and the location of the house to be connected. 3. The customer gives to the commercial employee the customer’s identification papers and the location of the house to be connected.. * 4. The commercial employee checks if the customer record is not written off and the customer record exists and the installation exists 5. If the customer record is not written off * 6. Then 7. If (the customer record exists) 8. Then 9. The commercial employee creates the customer record 10. If (the installation exists) 11. Then 12. The commercial employee asks to the customer to sign the contract. 13. The commercial employee asks to the customer to pay the deposit. 14. The customer gives to the commercial employee the signed contract. 15. The customer gives to the commercial employee the money for deposit. * 16. The commercial employee gives back to the customer the customer’s identification papers. 17. The commercial employee checks if the meter exists. 18. If (the meter exists) 19. Then 20. The commercial employee sends to the technical employee a service order for meter connection.
Describing Business Processes with a Guided Use Case Approach
357
21. A technical employee performs the connection to the network. 22. The technical employee informs the commercial employee that the meter connection is done. 23. Else 24. The commercial employee sends to the technical employee a service order for meter installation and a service order for meter connection. 25. A technical employee performs the meter installation and the connection to the network. 26. The technical employee informs the commercial employee that the meter installation and the meter connection are done. 27. Else 28. The commercial employee requests to technical employee to investigate the site. 29. The technical employee performs investigation 30. The technical employee informs the commercial employee that the investigation is done. 31. The commercial employee calculates price 32. The commercial employee asks to the customer to pay for the installation. 33. If the customer pays the commercial employee for installation * 34. Then 35. The commercial employee sends to technical employee a service order for installation. 36. The technical employee performs installation 37. The technical employee informs the commercial employee that the installation is done. 38. The customer is notified by the commercial employee that the installation is done. 39. If the customer asks to the commercial employee a connection to the network * 40. Then 41. The commercial employee asks to the customer to sign the contract. 42. The commercial employee asks to the customer to pay the deposit. 43. The customer gives to the commercial employee the signed contract. 44. The customer gives to the commercial employee the money for deposit. * 45. The commercial employee sends to the technical employee a service order for meter installation and a service order for meter connection. 46. A technical employee performs the meter installation and the connection to the network. 47. The technical employee informs the commercial employee that the meter installation and the meter connection are done. 48. The commercial employee sends to the customer a copy of the contract. 49. The customer is informed by the commercial employee that the connection to the network is done.
358
S. Nurcan, G. Grosz, and C. Souveyet
Final states : The customer is connected to the network. The customer has the customer’s identification papers. The customer has a copy of the contract. Sometimes, EC has a new customer. EC has a contact for this customer. EC has the network extended with a meter connection. Sometimes, EC has the network extended with a meter installation. Sometimes, EC has the network extended with an installation. Fig. 9. Version 4 of the Normal episode
3.6 Completing the Use Case Specification with « System Interactions » Now that we have integrated all scenarios describing the interactions between the human agents within the use case specification, we shall concentrate on the interactions between the agents of the organization and an automated system that shall support the business process, what is called « system interactions » in section 1, figure 1. To this end, we have defined a set of completion rules which aim at querying the use case author about the requirements for a computerized system that can support the performance of the process. The rules concentrate on the both the communication and internal actions. For each communication action, the author is asked if the communication is supported by the system. If this is so, the author completes the action accordingly. Similarly, for each internal action, the use case author is asked if the action is supported in some way by the system. This also leads to the emergence of new requirements for the system internal and to the completion of the system interactions. Using these rules, the dialogue leads to complete the normal episode with system interactions. The result is shown in figure 10 (EC IS stands to the Electricity Company information system, it is the information system that supports the EAF business process). 1. The customer requests the commercial employee for a connection to the network. 2. The commercial employee asks to the customer the customer’s identification papers and the location of the house to be connected. 3. The customer gives to the commercial employee the customer’s identification papers and the location of the house to be connected.. * 4. The commercial employee requests to EC IS if the customer record is not written-off and the customer record exists and the installation exists 5. EC IS checks if the customer record is not written-off and the customer record exists and the installation exists 6. EC IS informs the commercial employee if the customer record is not written-off and the customer record exists and the installation exists 7. If the customer record is not written-off * 8. Then 9. If (the customer record exists) 10. Then 11. The commercial employee requests to EC IS to create the customer record 12. EC IS creates customer record 13. EC IS acknowledges the creation of customer record to the commercial employee 14. If (the installation exists)
Describing Business Processes with a Guided Use Case Approach
359
15. Then 16. The commercial employee asks to the customer to sign the contract. 17. The commercial employee asks to the customer to sign the contract to pay the deposit.* 18. The customer gives to the commercial employee the signed contract. 19. The customer gives to the commercial employee the money for deposit.* 20. The commercial employee gives back to the customer the customer’s identification papers. 21. The commercial employee requests to EC IS if the meter exists. 22. EC IS checks if the meter exists 23. EC IS informs the commercial employee if the meter exists 24. If (the meter exists) 25. Then 26. The commercial employee requests to EC IS to send to the technical employee a service order for meter connection. 27. EC IS sends to the technical employee a service order for meter connection 28. A technical employee performs the connection to the network. 29. The technical employee informs the EC IS that the meter connection is done. 30. EC IS informs the commercial employee that the meter connection is done 31. Else ...... Fig. 10. Version 5 of the normal episode completed with system interactions
4. Conclusion Activities, such as business process engineering, business process re-engineering or business process improvement call for accurate description of business processes. Our proposal is about an approach supporting the construction of BP specifications. A BP specification takes the form of a use case comprising information about the context of the BP, the interactions between the agents involved in the BP, the interactions of these agents with a computerised system supporting the BP and a set of internal system requirements. We propose to guide the construction of a use case specification for BP using textual scenarios. A scenario describes, in natural language, interactions between different agents (i.e. a service requester and some service providers) and internal actions involving a single agent. An interaction is expressed as a communication action between two agents. A use case specification integrates all normal and exceptional scenarios describing a BP. We use the case grammar presented in [21] to analyse and to extract the semantics of textual scenarios. On top of the linguistic approach, the construction of use case specification is guided. Guidance is based on use case model knowledge and takes the form of rules which encapsulate knowledge about the use case model concepts in order to facilitate (1) the completion of scenarios, (2) emergence of other normal or exceptional scenarios and (3) integration of scenarios into a complete use case
360
S. Nurcan, G. Grosz, and C. Souveyet
specification. The guided process was illustrated with a real case study dealing with the business process "Electricity Application Fulfilment" borrowed to an Electricity Company. In this paper, we focused on the interactions between the organisational agents of the studied business process, and their interactions with the automated system supporting the BP. Our current work consists in extending the guidance rules to support the emergence of system internal requirements on one hand, and system contextual requirements on the other hand. The former relates to internal system objects and behaviour whereas the latter deals with the context in which the business process takes place (weaknesses, opportunities, non functional requirements, etc.). Meanwhile, we are completing the current PROLOG implementation to handle the entire set of rules presented in this paper. References 1. J.M. Caroll, The Scenario Perspective on System Development, in J.M. Carroll (ed.), Scenario-Based Design: Envisioning Work and Technology in System Development (1995). 2. A. Cockburn, Structuring use cases with goals, Technical report, Human and Technology, 7691 Dell Rd, Salt Lake City, UT 84121, HaT.TR.95.1, http://members.aol.com/acocburn/papers/usecases.htm (1995). 3. B. Dano, H. Briand, F. Barbier, A Use Case Driven Requirements Engineering Process, In Third IEEE International Symposium On Requirements Engineering (RE'97), Antapolis, Maryland (IEEE Computer Society Press, 1997). 4. E. Dubois, P. Heymans, M. Jarke, K. Phol, K. Weidenhaupt, A. Sutcliffe and N.A.M. Maiden, Integration of Scenario Representations: Formalisation and Examples, ESPRIT Reactive Long Term Research Project 21.903 CREWS, Deliverable W4: Knowledge Representation Working Group (1997). 5. T. Erickson, Notes on Design Practice: Stories and Prototypes as Catalysts for Communication, in John M. Carroll (ed.), Scenario-Based Design: Envisioning Work and Technology in System Development (John Wiley and Sons, 1995) 37-59. 6. C. Fillmore, The Case For Case, in Holt, Rinehart and Winston (eds.), Universals in Linguistic Theory (Bach & Harms, 1968) 1-90. 7. J.D. Gould, How to design usable systems, in M. Helander (ed.), Handbook of HumanComputer Interaction (Elsevier Science, 1988) 757-790. 8. Hammer M. Champy J., "Re-engineering the corporation : a manifesto for business revolution", Harper Collins publishers, inc., New York, 1993. 9. I. Jacobson, The Use Case Construct in Object-Oriented Software Engineering, in John M. Carroll (ed.), Scenario-Based Design: Envisioning Work and Technology in System Development (John Wiley and Sons, 1995) 309-336. 10. I. Jacobson, M. Ericsson and A. Jacobson, The Object Advantage, Business Process Reengineering with Object Technology (Addison-Wesley Publishing Company, 1995). 11. M. Jarke, K. Pohl, P. Haumer, K. Weidenhaupt, E. Dubois, P. Heymans, C. Rolland, C. Ben Achour, C. Cauvet, J. Ralyte, A. Sutcliffe, N.A.M. Maiden and S. Monicha, Scenario Use in European Software Organisations - Results from Site Visits and Questionnaires, ESPRIT Reactive Long Term Research Project 21.903 CREWS, Deliverable W1: Industrial problem capture Working Group (1997).
Describing Business Processes with a Guided Use Case Approach
361
12. J. Karat, Scenario Use in the Design of a Speech Recognition System, in John M. Carroll (ed.), Scenario-Based Design: Envisioning Work and Technology in System Development (John Wiley and Sons, 1995) 109-135. 13. K. Koskimies and H. Mossenbock, Scene: Using Scenario Diagrams and Active Text for illustrating Object-Oriented Programs, in Proc. of ICSE-18 (1995) 366-375. 14. M. Kyng, Creating Contexts for Design, in John M. Carroll (ed.), Scenario-Based Design: Envisioning Work and Technology in System Development (John Wiley and Sons, 1995) 85-107. 15. R.L. Mack, Discussion : Scenarios as Engines of Design, in John M. Carroll (ed.), Scenario-Based Design: Envisioning Work and Technology in System Development (John Wiley and Sons, 1995) 361-387. 16. J. Nielsen, Scenarios in Discount Usability Engineering, in John M. Carroll (ed.), Scenario-Based Design: Envisioning Work and Technology in System Development (John Wiley and Sons, 1995) 59-85. 17. C. Potts, K. Takahashi and A.I. Anton, Inquiry-based Requirements Analysis, in IEEE Software 11(2) (1994) 21-32. 18. J.C.S. do Prado Leite, G. Rossi, F. Balaguer, A. Maiorana, G. Kaplan, G. Hadad and A. Oliveros, Enhancing a requirements baseline with scenarios, In Third IEEE International Symposium On Requirements Engineering (RE'97), Antapolis, Maryland (IEEE Computer Society Press, 1997) 44-53. 19. B. Regnell, K. Kimbler and A. Wesslen, Improving the Use Case Driven Approach to Requirements Engineering, in the Second IEEE International Symposium On Requirements Engineering, York, England (I.C.S. Press, March 1995) 40-47. 20. S.P. Robertson, Generating Object-Oriented Design Representations via Scenarios Queries, in John M. Carroll (ed.), Scenario-Based Design: Envisioning Work and Technology in System Development (John Wiley and Sons, 1995) 279-308. 21. C. Rolland, C. Ben Achour : Guiding the construction of textual use case specifications, in the Data & Knowledge Engineering Journal, 25(1-2) Special Jubilee issue, March 1998, 125-160. 22. M.B. Rosson and J.M. Carroll, Narrowing the Specification-Implementation Gap, in John M. Carroll (ed.), Scenario-Based Design: Envisioning Work and Technology in System Development (John Wiley and Sons, 1995), 247-278. 23. J. Rumbaugh, M. Blaha, W. Premerlani, F. Eddy, and W. Lorensen, Object-Oriented Modeling and Design, (Prentice Hall, 1991). 24. J. Rumbaugh and G. Booch, Unified Method, Notation Summary Version 0.8 (Rational Software Corporation, 1996). 25. K. Weidenhaupt, K. Pohl, M. Jarke, P. Haumer, Scenario Usage in System Development : a Report on Current Practice, ESPRIT Reactive Long Term Research Project 21.903 CREWS, Deliverable W1-D2, (1997). 26. J. Whiteside, J. Bennett and K. Holtzblatt, Usability Engineering : Our experience and evolution, in M. Helander (ed.), Handbook of Human-Computer Interaction (Elsevier Science, Amsterdam, 1988) 791-818. 27. R. Wirfs-Brock, Designing Objects and their Interactions: A Brief Look at Responsibilitydriven Design, in John M. Carroll (ed.), Scenario-Based Design: Envisioning Work and Technology in System Development (John Wiley and Sons, 1995) 337-360.
362
S. Nurcan, G. Grosz, and C. Souveyet
Appendix The use case specification is composed of the contextual information shown below, one normal episode shown in figure 9 and four exceptional episodes. Contextual information Use case name Initiating agent Goal Initial states
Electricity Application Fulfilment A customer of the Electricity Company (EC) To connect the customer to the company network Customer is present in the EC office with a completed application form.
Exceptional episode of the use case Electricity Application Fulfilment Name : WrittenoffCustomer Occurrence condition : When (the customer record is not written-off). Where : the action 5 of the NormalCase episode. Action : 1. The commercial employee informs the customer that the customer record is written-off and the connection to the network can not be done. 2. The commercial employee gives back to the customer the customer’s identification papers. Final states : The customer is not connected to the network. The customer has the customer’s identification papers.
Exceptional episode of the use case Electricity Application Fulfilment name : NoIdentificationPaper occurrence condition : When (the customer gives to the commercial employee the customer’s identification papers and the location of the house to be connected ). Where : the action 3 of the NormalCase episode. action : 1. The commercial employee informs the customer that the connection to the network can not be done without customers identification paper’s. Final states : The customer is not connected to the network. Exceptional episode of the use case Electricity Application Fulfilment name : NetworkConnectionAborted occurrence condition : When (the customer gives to the commercial employee the signed contract and the money for deposit) OR (the customer pays the commercial employee for installation). Where : the action 15 and a year after action 32 of the NormalCase episode. action : 1. The commercial employee informs the customer that the connection to the network can not be done without payment. Final states : The customer is not connected to the network. The customer has the customer’s identification papers. Exceptional episode of the use case Electricity Application Fulfilment name : InstallationOnly occurrence condition : When (the customer asks to the commercial employee a connection to the network) OR (the customer gives to the commercial employee the signed contract and the money for deposit). Where : the action 43 and 6 months after action 38 of the NormalCase episode. action : 1. The commercial employee informs the customer that the network connection request is aborted. Final states : The customer is not connected to the network. The customer has the customer’s identification papers. , EC has the network extended with an installation.
Building Quality into Case-Based Reasoning Systems 1
Igor Jurisica1 and Brian A. Nixon2 University of Toronto, Faculty of Information Studies 140 St. George St., Toronto, ON M5S 3G6, Canada jurisica@ s.utoronto.ca 2 University of Toronto, Dept. of Computer Science Toronto, ON M5S 3H5, Canada [email protected]
Abstract. Complex decision-support information systems for diverse
domains need advanced facilities, such as knowledge repositories, reasoning systems, and modeling for processing interrelated information. System development must satisfy functional requirements, but must also systematically meet global quality factors, such as performance, con dentiality and accuracy, called non-functional requirements (NFRs). To build quality into an important class of decision support systems, case-based reasoning (CBR) systems, this paper presents \QualityCBR," a goal-oriented, knowledge-based approach for systematically dealing with NFRs for CBR systems. With the idea that similar problems have similar solutions, CBR systems store cases (problems with solutions) and solve new problems by retrieving and reusing similar past cases. QualityCBR integrates existing work on CBR and NFRs. It helps developers state and re ne NFRs, consider tradeo s, make decisions and evaluate their impact on NFRs. We illustrate the approach in a complex medical domain, in vitro fertilization, where CBR suggests therapy for patients, predicts the probability for successful pregnancy, and determines patient's characteristics that can improve pregnancy rate.
1 Introduction Complex information systems in both the public and private sectors need a number of advanced facilities, including decision support systems (DSSs), repositories, reasoning systems, and facilities for modeling and processing large amounts of complex information. Many DSSs have been built for individual domains in an ad hoc manner. However, to e ectively build families of DSSs for complex domains, such as medical, governmental or industrial applications, we need a systematic knowledge-based approach to: (1) empower expert user to make e ective decisions using a DSS, and (2) address concerns for quality and performance requirements. Case-based reasoning (CBR) systems [25] are an important class of DSSs that represent experiences (problems with solutions) as cases. Cases are used for solving new problems by accessing past cases and comparing their similarity to a given problem. In this paper we use a generic CBR system called TA3 B. Pernici and C. Thanos (Eds.): CAiSE’98, LNCS 1413, pp. 363-380, 1998. Springer-Verlag Berlin Heidelberg 1998
364
I. Jurisica and B.A. Nixon
(pronounced tah-tree) to build a complex medical DSS, which can be used to advise physicians who prescribe treatment plans for in vitro fertilization (IVF) patients [24]. CBR systems must meet functional requirements, including retrieving past cases, selecting and reasoning about relevant ones, interactively exploring cases, and adapting them to produce a solution, which is then evaluated. In addition, CBR and other large and complex information systems must meet non-functional requirements (NFRs or quality requirements), which are global requirements for quality factors such as performance, accuracy and con dentiality. NFRs are important for the success of complex systems. In the case of medical systems, con dentiality is crucial. Dealing with NFRs systematically is dicult, because a developer must consider not only requirements, but also implementation alternatives and tradeo s. In addition, requirements can be in con ict with each other (e.g., it may not be possible to have both expressive representation and fast access time). There are many implementation alternatives, with impact on di erent NFRs (e.g., having entry clerks type information twice might improve accuracy, but decrease user friendliness). Decisions are interrelated, with a global impact on the target system. For these reasons, one can't simply use \canned alternatives" to meet the quality requirements. Instead, we use an approach where the developer considers the characteristics of a particular system being developed and application needs in a systematic way. This provides a process that helps producing customized systems that meet quality requirements. Simple applications can usually be built in an ad hoc manner, and dealing with requirements may not be dicult. However, a distinguishing aspect of large and complex information systems, whether medical, governmental or industrial, is that characteristics including data, algorithms, domains, requirements, priorities and workload must all be considered. Furthermore, these characteristics interact in complex ways. Hence it is important to deal with them in a systematic way. We deal with the complexity of these kinds of systems by: using a knowledgebased approach which catalogues expertise, o ering competent and ecient CBR facilities, and using a structured approach to deal with NFRs. These facilities are combined in our approach, called QualityCBR. To provide a development process that addresses NFRs for CBR, and is goaloriented, systematic developer-directed and qualitative, we draw on the \NFR Framework" [4, 6, 31]. The NFR Framework supports this process of building quality in to a system interactively, while considering NFRs throughout the development process. Quality requirements are treated as goals to be systematically achieved during the design and development process. The NFR Framework, with its associated tool, helps a developer state and re ne NFRs, consider design tradeo s, justify decisions and evaluate their impact on NFRs, while giving the developer control of the development process. To deal with performance requirements, we draw on the Performance Requirements Framework [33, 34, 35], a specialization of the NFR Framework.
Building Quality into Case-Based Reasoning Systems
365
The factors which must be considered during development may all change during a system's lifetime. This greatly increases the complexity of development, and further motivates the need for a systematic approach. By using a knowledgebased approach, and by drawing on the NFR Framework's facilities for dealing with change [7], we can systematically deal with change. There are two possible combinations of techniques for CBR and NFRs: (1) using CBR to support reasoning about and reuse of NFRs, and (2) using NFRs to systematically build quality into a CBR system. This paper addresses the latter issue, using the QualityCBR approach. In particular, we describe the process of using QualityCBR and providing catalogues that deal with alternative implementations. QualityCBR draws on a exible knowledge representation language for information systems | Telos [30], relevance assessment [20], similarity-based retrieval algorithm [22], and the NFR Framework's goal-oriented, qualitative approach [31]. In addition, QualityCBR uses knowledge discovery algorithms [1, 24] and data model implementation experience [36]. QualityCBR is applied to a complex medical DSS for IVF practitioners TA3 [24]. During the development of the system we considered some NFRs, albeit in an ad hoc manner. We show how a developer could use QualityCBR to systematically build a CBR system for IVF by addressing NFRs such as performance { \Select relevant cases quickly", con dentiality { \Store patient records securely", and informativeness { \Display results informatively". We also consider the impact of some changes.
2 The QualityCBR Approach This section presents the elements of the QualityCBR approach, for addressing non-functional requirements of case-based reasoning systems. Traditionally, CBR system were developed for a speci c application. The presented work aims at de ning a generic framework that is adaptable for di erent domains, while ensuring that both functional and non-functional requirements are systematically met.
2.1 Case-Based Reasoning This section describes principles of case-based reasoning (CBR) and a particular prototype TA3. Our aim is a exible system that can be applied to various domains, without sacri cing system performance. We consider system performance as a quality of solution and its timeliness. A case-based reasoning approach [25] relies on the idea that similar problems have similar solutions. Facing a new problem, a CBR system retrieves similar cases stored in a case base and adapts them to t the problem at hand. Informally, a case comprises an input (the problem), an output (the solution) and feedback (an evaluation of the solution). CBR involves the process of: (1) Accepting a new problem description; (2) Retrieving relevant cases from a case base (past problems with similar input); (3) Adapting retrieved cases to t the input
366
I. Jurisica and B.A. Nixon
problem and nding a solution to it; and (4) Evaluating the solution (producing feedback for the case). Considering the above CBR cycle, one can say that the more similar the cases are, the less adaptation is necessary, and consequently, the proposed solution may be more correct. Then, an important task is how to measure case relevance (similarity or closeness) to guarantee retrieving only highly relevant cases, i.e., cases that are similar according to speci ed criteria, and thus can be useful in solving the input problem in a particular context. Thus, we need a variable-context similarity assessment. In many processes, it is better to retrieve fewer cases, or none, than to retrieve less useful cases that would result in a poor solution. But similarity of cases is only one measure of system quality. It is also important that the solution be provided quickly. It should be noted that the tradeo between closeness and timeliness of a solution depends on requirements of a particular application [19]. For these reasons we use a variable-context similarity assessment and case base clustering as described next. TA3 is a CBR system, which uses a variable-context similarity-based retrieval algorithm [22] and a exible representation language. Knowledge must be represented in a form appropriate for the intended user, and the representation should be rich enough to support complex, yet ecient processing [23]. Cases are represented as a collection of attribute-value pairs. Individual attributes are grouped into one or more categories [22]. Categories bring additional structure to a case representation. This reduces the impact of irrelevant attributes on system performance by selectively using individual categories during matching. As a result, we get a more exible reasoning system [19], a more comprehensible presentation of complex information [20], improved solution quality [24], and improved scalability [23]. During the CBR process, we want to handle partial as well as exact matches. We have a partial matching when attribute values of one case match only a subset of values of another case. In order to retrieve and control both exact and partial matching, a view of a case, called a context, is de ned. Thus, a case to be interpreted in a given context. By controlling what constitutes a partial match, context speci es important attributes and how \close" an attribute value must be. We say that a case satis es (or matches) a particular context, if for each attribute speci ed in the context, the value of that attribute in the case satis es the constraint [22]. Thus, the matching process can be described as a constraint satisfaction problem [40]. The quality of the matching process is measured by the closeness of retrieved cases [22], timeliness of the answer [23], and adaptability of the suggested solution [26]. Ortega has shown that partial m-of-n matches improve performance if m is reasonably selected [37]. Our approach of representing cases as sets of Telosstyle categories [30], each comprising a set of tuples, allows for multiple levels of m-of-n matching. Thus, important attributes may require n-of-n matches for a given category, and less important attributes may allow for k-of-n matches (k < n). The problem is to nd these attribute groupings, i.e., a context that speci es which attributes are needed for accurate prediction, and what range or
Building Quality into Case-Based Reasoning Systems
367
similarity should be allowed for attribute values. This knowledge can be automatically discovered [24] and can be used for case base clustering by: (1) appropriately grouping attributes into categories (clustering of attributes); (2) discovering what values are \close" for particular attributes (clustering of attribute values); and (3) structuring the case base into clusters of relevant cases (clustering of cases).
2.2 Handling Non-Functional Requirements The NFR Framework [4, 31] helps a developer represent and use key concepts about NFRs (e.g., security and performance), the particular domain (e.g., IVF), and development expertise, (e.g., CBR, databases and system development). Being in uenced by work in DSSs [28], the NFR Framework maintains a concise and structured development graph whose components record the developer's goals, decisions and design rationale. The developer states a set of NFRs for the system, which are represented as goals that are considered throughout the system development process. In trying to meet the requirements, developers are helped in choosing among design alternatives, which are organized in a catalogue of methods. Partial positive or negative relationships among goals are recorded as qualitative link types. Knowledge of design tradeo s is arranged in a catalogue of correlation rules. After decisions are made, the NFR Framework uses its evaluation algorithm to help the developer determine if overall goals have been met. Section 3.2 presents the components of the NFR Framework in more detail, and illustrates their use. The NFR Framework has been previously applied to information systems in several domains, in both the public and private sectors (e.g., health insurance, banking and government systems) [5, 7]. Its approach can be specialized to deal with a number of NFRs, such as performance [33, 34, 35], accuracy [4] and security [3]. For performance, for example, we represented principles for building good response time into systems [39] and arranged information system implementation knowledge using a layering approach [17] based on data model features, to reduce the number of issues considered at a time. The \NFR Assistant" prototype tool [4], provides support to a developer using the NFR Framework, by providing catalogues of concepts and methods, aiding the construction and evaluation of development graphs, and keeping track of correlations. The tool draws on the ConceptBase system [18] which uses the Telos [15, 30] knowledge representation language. A specialization of the tool, the Performance Requirements Assistant [34, 35], o ers catalogues of concepts and techniques for treating performance requirements, using other Telos-based knowledge base management tools,1 but o ers only a subset of the functionality of the NFR Assistant. 1
M. Stanley's Telos sh and B. Kramer's RepBrowser, at the University of Toronto.
368
I. Jurisica and B.A. Nixon
2.3 Cataloguing CBR and NFR Knowledge The QualityCBR approach organizes knowledge about issues and techniques for CBR and NFRs. These knowledge bases, represented in Telos, serve as a basis for recording experts' knowledge and are used during system development, They help a user to satisfy NFRs (such as performance and con dentiality), e ectively use CBR techniques (e.g., knowledge representation, retrieval), and consider particular characteristics of the system under development (e.g., workload, con dentiality concerns). Clustering
Partial
Full
DK for KDD DK for EBL DK for DB
Knowledge Domain Data Knowledge Discovery (DK) (KDD)
Explanation Based Learning (EBL)
Database Techniques (DB)
Fig. 1. A Catalogue of Clustering Techniques for CBR. Some steps, typically done early in using the approach, involve de ning and organizing a variety of types of knowledge applicable to the system under development. This produces a number of catalogues of concepts:
{ Concepts about a particular class of reasoning systems (e.g., CBR), such
as components of the CBR system, problem decomposition techniques and implementation alternatives. Figure 1 shows a sample catalogue for implementation alternatives for clustering techniques. Specialized catalogues draw on combinations of aspects, e.g., domain knowledge for knowledge data discovery. { Concepts about particular NFRs (e.g., performance and security). For example, a terminology of performance concepts is made available, along with a catalogue which shows the impact of implementation techniques on time and space goals [34]. { Concepts about the particular application domain, e.g., IVF: descriptions of processes (e.g., a cycle of patient treatment) and workload (e.g., number of patients). { Generic concepts associated with the NFR Framework, e.g., de nitions of the components of development graphs which record developers' decisions.
Building Quality into Case-Based Reasoning Systems
369
3 Illustrating the QualityCBR Approach This section shows the use of QualityCBR's components and cataloguing to support several NFRs for the IVF domain. Section 3.1 presents the domain of our study of the IVF system. We consider top level non-functional requirements for an IVF system, which could be stated by a developer, involving: performance { patient records must be retrieved and analyzed quickly (Section 3.2), and con dentiality { records must be stored securely (Section 3.3). In addition, the system should be robust and user-friendly (Section 3.4).
3.1 Functional Requirements in the IVF Domain
In vitro fertilization (IVF) is an example of a complex medical domain, where
DSS can be used to suggest the hormonal treatment and to support research [24]. Individual patients respond to the treatment di erently. A patient's response and the pregnancy rate depends on many attributes. While experienced doctors can use their knowledge to suggest a treatment for a patient, it is dicult for them to perceive trends and make informed decisions to optimize success rates for each individual infertile couple. This is especially a concern when knowledge about in uencing factors changes. Prediction of the likelihood of pregnancy involves suggestion of a treatment. This is performed in two stages. First, given initial information about the patient (diagnosis, previous treatment history, etc.) the task is to nd similar patients from the case base and make a suggestion of how to treat the current patient to increase the probability of successful pregnancy. This includes nding all relevant cases, and considering retrieved cases with pregnancy as successful examples and retrieved cases without pregnancy as negative cases. An adaptation process uses this information to suggest values for remaining attributes in the current case, namely how long the patient should be stimulated and what amount of the hormones should be used. Second, after the initial treatment is completed, additional attributes are available (patient's responsiveness to the hormonal stimulation). The task is then to predict the outcome of the whole treatment, i.e., to predict likelihood values for pregnancy and for unsuccessful cases. The prediction task can also be considered as an optimization problem: for a given patient minimize the amount of hormonal therapy required, without compromising the outcome. Knowledge discovery is used to nd regularities in the case base by using knowledge-miningtechniques, as well as to suggest missing data. Here, physicians have no particular case in mind, however, they may consider the whole knowledge base or only certain cases. Knowledge mining in TA3 involves nding a context in which a particular group of cases is considered similar. The user has the ability to specify a threshold, which controls the quality and the quantity of discovered information [24]. Considering that each patient is described by about a hundred attributes [24], that there are about 600 patients per year per clinic and that there are about 300 IVF clinics in North America [29], the problem is not simple. Moreover, IVF information is more sensitive than general medical information and the
370
I. Jurisica and B.A. Nixon
complex IVF process involves various professionals, which need to access part or whole information about the patient. IVF has relevance to both the public and private sectors. In the Province of Ontario, Canada, for example, publiclyfunded health insurance covers the cost of IVF for certain forms of infertility, e.g., tubal blockage, while others are not covered, and are handled by private clinics.
3.2 Dealing with Performance Requirements
We now show how performance requirements for the IVF domain are handled using QualityCBR. We also describe components of the NFR Framework used in QualityCBR. System performance is an important factor for complex applications. Good performance includes fast response time and low space requirements. For the IVF system, a developer might state that one important goal is to have fast response time when accessing patient records, for reasoning as well as case updating. This requirement is represented as a goal: Time[Patient Records and Reasoning], as shown in Figure 2. Time is the sort of the goal (i.e., the particular NFR concept, addressed by the goal) and [Patient Records and Reasoning] is the parameter (i.e., the subject) of the goal. (The entries within circles will be discussed below.) Another main goal is to have fast response time for reasoning operations done by researchers, represented by Time[Research Reasoning].
Claim["Aid Doctor"] And
+
NFR Goal
Satisficed Goal
Satisficing Goal
Neutral Goal
Argument
Denied Goal
!
Time [Update]
++
Legend
Time [Research Reasoning]
Time [Patient Records and Reasoning]
!
--
Time [Prediction]
+ + ++ -- ++ --
Time [Discovery]
Evaluation link Correlation link
Priority goal
++ Very Positive Link + Positive Link --
Negative Link Very Negative Link
No Partial Full Clustering Clustering Clustering
Fig. 2. Dealing with Performance Requirements for Reasoning. Using methods and catalogues of knowledge (for performance, CBR, IVF, etc.), goals can be re ned into more specialized goals. Here, the developer used knowledge of the IVF domain to re ne the time goal for patient information into two goals, one for good response time for updating patient records and the other for good response time for the retrieval and decision making process. These two o spring goals are connected by an And link to the parent goal. This means that
Building Quality into Case-Based Reasoning Systems
371
if both the goal for fast updates and the goal for fast prediction are accomplished then we can say that the parent goal of fast access to patient records will in some sense be accomplished. The NFR Framework takes a qualitative, \satis cing" approach, in which goals are more-or-less met, although they may not be satis ed in an absolute sense [38]. Similarly, the goal of good response time for research reasoning can be re ned into a goal of fast response for the \discovery" process which searches patient records for patterns. Here, the parent has one o spring, connected by a positive (\+") link, which indicates that accomplishing the o spring will contribute positively towards accomplishing the parent goal. Other types of relationships can be shown by other link types (see Figure 2). In building quality into a system, it is important to identify priorities. For the case of building performance in, we should identify time-critical operations as well as those which dominate the workload [39]. Here, we identify the prediction operation as being time-critical (indicated by \!"), and provide a reason using domain knowledge: it is important to aid the doctor by quickly suggesting a treatment. This is an example of recording design rationale [28] { the reasons for decisions { using the NFR Framework's arguments. As part of the development graph, arguments are available when making further decisions and changes. It is important to note that the developers use their expertise to determine what to re ne, how to re ne it, to what extent to re ne it, as well as when to re ne it. The NFR Framework and its associated tool help the developer, do some consistency checking, and keep track of decisions, but it is the developer who is in control of development process.
Implementation Alternatives.
In moving towards a target system, one must consider implementation alternatives for case base clustering, which appropriately groups attributes, their values, and relevant cases together. The main concern for clustering is with the storage of patient records, which besides general patient information (name, address, etc.) consist of attribute-value pairs describing the diagnosis of infertility, previous and current treatments, the result, etc. E ective storage of this information facilitates the various CBR operations, because individual pieces of information have di erent importance and di erent e ects on the treatment and on the overall outcome. Currently, the information is recorded in a paper-based form with general patient information being sent to a central hospital computer. A computerized IVF case base is populated in a batch process. Many of the implementation alternatives (shown as dark circles in Figure 2) will be drawn from appropriate catalogues. Implementation alternatives for the following clustering operations must be considered: { Storage and update. In the IVF application, data entry and updates have the form of lling in blanks, either selecting a value from a pick-lists or typing it. Considering the amount of data in one clinic, storage and update are not major problems. However, taking into account possible extensions, e.g.,
372
I. Jurisica and B.A. Nixon
linking several IVF clinics in a network to share their case bases, it is useful to note this requirement. { Prediction. A doctor uses the system to suggest a hormonal therapy for the current patient (see Section 3.1). It is important that the accuracy of predicted information is within reasonable bounds and a solution is provided swiftly. There is a relationship between accuracy, time and space: the more cases are stored, the more accurate solutions can be provided, but the longer it takes to nd cases relevant to a given problem. { Knowledge discovery. Treatment protocols can be improved by using knowledge discovery [24]. Discovered knowledge is used to organize attributes into categories, and cases into clusters (equivalence classes). The above considerations a ect implementation alternatives (\satis cing goals") for case base clustering: (1) the system may not use any clustering; (2) it may use full clustering; or (3) an hybrid, a partial clustering scheme can be deployed; further variations of clustering from the methods catalogue can be considered (see Figure 1). Without clustering, updates are faster, as data need not be reorganized; however, prediction is slower as there is no clustering to aid the retrieval process. Thus, at the bottom left of Figure 2, No Clustering is shown to have a positive impact on update time, and a negative impact on prediction time. Full clustering is done by knowledge discovery: it speeds up prediction, but hinders update time. No and full clustering each slow down at least one of the three operations. The developer can formulate alternatives which reduce or avoid this problem. Partial clustering may start with cases clustered using domain knowledge, but may subdivide certain clusters into more detailed groups. Its main advantage is that it speeds up all three operations, instead of slowing any of them. However, no clustering is better (\++") than partial for update, and full clustering is better than partial for retrieval. Thus, partial clustering o ers intermediate performance for some operations, but avoids p bad performance for all of them. As a result, partial clustering is selected (\ ") over the unchosen (\") alternatives. Note, that an IVF facility that does not support research may give low priority to performance for knowledge discovery. Since the hormonal therapy suggestion would have high priority, full clustering would be selected.
Evaluating Goal Accomplishment.
After decisions are made, the developer can determine their overall impact on the system's requirements. The developer is aided by the NFR Framework's semi-automatic evaluation algorithm, which examines the development pgraph, generally bottom-up. It starts with implementation decisions to accept (\ ") or reject (\") alternatives (shown in dark circles at the bottom of Figure 2), Results then propagate upward along evaluation links. Evaluation assigns values (e.g., \p" or \") to parent goals based on the values of o spring goals, and and the relationships (link types, e.g., \+" or \-") between o spring and parent goals. For example, with a \+"plink type, meeting (\p") the o spring (e.g., Partial Clustering) helps meet (\ ") the parent; however, if the o spring
Building Quality into Case-Based Reasoning Systems
373
is denied (\", not achieved), the parent will be denied (\"). The \-" link type can be thought of as giving the parent the \opposite" of the o spring's label. Values from all applicable o spring propagate to a parent. Here, partial clustering helps quickly accomplish updating, presentation and discovery. During the process, the developer may step in, to determine propagated values. For example, if a parent goal received positive and negative values from di erent o springs, the developer is able to resolve the con ict using domain knowledge. It should be noted that not all goals can always be met, but performance can be enhanced if the priorities are accomplished [39]. As presented in Figure 2, the critical goal for prediction has been met. Since the update time goal was also met, the top goal for records and reasoning was met. As the discovery goal was met, the top goal for research reasoning was also met.
Dealing with Changes in Priorities. Let's consider four imaginary IVF clinics with di erent priorities: (1) fast update of records, (2) fast prediction, (3) both fast prediction and fast update are important, and (4) fast case base analysis (discovery). Depending on the priorities, we may adjust the solution of Figure 2 by choosing a di erent alternative. As a result, the rst clinic would not use clustering, the second would use full clustering, and the third and fourth clinics would achieve their requirements by deploying partial (hybrid) clustering. This is an example of reusing an existing development graph, which uses the NFR Framework's components to capture decisions and rationale, as a resource for rapid analysis of the impact of change upon the achievement of NFRs [7]. In addition, we have used domain knowledge, priorities and performance catalogues to produce customized solutions which meet needs of a particular organization.
3.3 Security Requirements Security is an important factor, especially in medicine, and IVF is a particularly sensitive application. Security includes such concepts as integrity, con dentiality and availability [4], whose combination is used in a generic methodology for medical data security [14]. For the IVF clinic, we identi ed two primary goals (top part of Figure 3): (1) The physical integrity of gametes of the patient is extremely crucial (indicated by \!!"). (2) The con dentiality of patient data should be maintained. A third goal is to maintain the professional integrity (reputation) of the doctor (researcher).
Physical Integrity of Patient Material. The crucial concern is that a patient's gametes must not be mistaken for someone else's. Thus, accurate identi cation of gametes strongly contributes to physical integrity. This can be accomplished either by using patient's name or an identifying number. Using only a number might contribute to con dentiality; for example, the lab technician, who deals with gametes, but not directly
374
I. Jurisica and B.A. Nixon Integrity [Patient] !!
Integrity Confidentiality [Doctor] [Patient Info]
+
+
!
++ And Confidentiality [Non-Identifying Info]
Confidentiality [Identifying Info] ! Accurate ID [Patient !! gametes]
Confidentiality [Identifying Info In Lab] !
--
Number Only
And
+
!
Confidentiality [Identifying Info Outside Lab] --
Name In Lab
Name Outside Lab
Number Outside Lab
Fig. 3. Dealing with Security Requirements for an IVF Clinic. with patients, could in principle use only numbered dishes without knowing patient's name. However, this could increase the chance of confusing gametes of two patients, which must be avoided. Instead, the lab labels dishes with gametes using the patient's name, which is only made available to authorized personnel, including the technician. The analysis is shown in the lower left of Figure 3. It's interesting to note the interaction between the goals of physical integrity of gametes, and the con dentiality of patient information, and the resolution of the con ict to bene t the higher-priority goal. In addition, to help meet both goals, the lab has a system of physical security. While this is not shown in the gure, it is important to note that measures taken outside the computer system can have an impact on the NFRs being considered.
Con dentiality of Patient Information. The IVF clinic records some basic identifying information about a patient (name, age, hospital number, etc.), a record of the patient's visits during a treatment cycle, treatments made, and observations. In addition, the central hospital computer maintains accounting records, which do not have the details of IVF treatment. Patient information is used for both tracking individual patients for treatment, and for reviewing groups of patients for research purposes. This dual usage complicates con dentiality considerations. Furthermore, researchers sometimes need to obtain further information about particular patients, hence the statistical research information must contain some patient identi ers. Clearly, access to medical data should be restricted to authorized persons, in relation to their status [13]. In the case of the IVF clinic, the mere fact that someone is an IVF patient is considered quite a personal matter, hence con dential [10]. The issue of security of statistical data is a complex one. According to [32]: \confusion still surrounds the question of whether privacy can be fundamentally
Building Quality into Case-Based Reasoning Systems
375
violated by statistics drawn from personal records". However, it was also shown that statistical information could provide detailed information about individuals [9]. The more information pieces are tied together the more identi able the individual is. Con dentiality of patient information must handle two goals: (1) information that identi es a patient and (2) information that does not (see Figure 3). In IVF domain, data can be used both for clinical treatment and for research. Thus, the goal of con dentiality of identifying information can be re ned by the developer to handle these situations. As discussed earlier, the patient's name will be used within the lab, to meet the overriding goal of integrity of gametes, which (along with the goal of con dentiality of records) will be aided by physical security measures. To reduce the risk of names being divulged to third parties, the patient's name should not be used outside the lab. Instead, an identi cation number (hospital number, sample number or user generated number [16]) is used.
Evaluating the Overall Impact of Decisions. Using a name within the lab helps accurately identify gametes, and maintain its physical integrity. The selective use of name and number provides con dentiality of identifying information, both inside and outside the lab. Meeting this critical goal contributes to the overall con dentiality of patient information. In turn, meeting both that con dentiality goal and the goal for physical integrity of gametes contributes positively to maintaining the professional integrity of the doctor (researcher). While we did not initially identify professional integrity as a main goal, it is very interesting to see that the result of our analysis using the NFR Framework was in harmony with the observation that the integrity of researchers is paramount [2].
3.4 Other NFRs Additional NFRs for the presented system include: (1) Robustness: the ability to gracefully handle a variety of situations; (2) User friendliness: providing the right degree of assistance to users; and (3) Informativeness: providing the right amount of information, appropriately arranged. Robustness concerns for the CBR system include: (1) reducing the e ect of irrelevant attributes on CBR so that the prediction accuracy does not degrade with an increased number of irrelevant attributes and presenting only attributes relevant to the task; (2) fault tolerance during data entry and reasoning. Thus, the goal for robustness of the system is re ned into goals for data integrity, robustness of reasoning and robustness of presentation (Figure 4, top left). Data integrity is important [14]. As suggested in [13], veri cation and validation of data completeness and accuracy is an additional measure ensuring data integrity. Thus, especially in the early stages of system development, all attributes available should be used. This allows for correlating the attributes, which can lead to identifying data integrity violations. However, if all attributes are also used in later stages, this would lead to problems with reasoning and
376
I. Jurisica and B.A. Nixon User
Robustness[System]
Data Integrity [System]
Friendliness[System]
And Robustness [Reasoning]
+ Robustness [Presentation]
-
+ + + + +
++
+
Use [RelevantAttrib.]
Use [AllAttrib.]
-
-
-
Syntactic Check [Data]
+
+
Early
Later
Informativeness[System] +
--
+
SemanticCheck [Data]
RetypeData Early
Later
Fig. 4. Dealing with Several NFRs for the System. presentation. Thus, only relevant attributes should be used in later phases. As described in Section 2.1, knowledge-discovery techniques can be used to locate features relevant to the problem solving task [24]. Using only relevant features improves exibility [20], accuracy [22], and eciency [23]. The e ect of this selective use of attributes contributes positively to the top goals of robustness and informativeness, both of which are accomplished, but user friendliness is not accomplished for the reasons described below. Generic relationships between NFRs and satis cing goals can be catalogued in advance as \correlation rules." These relationships can then be detected by the NFR assistant system, even if the developer has not explicitly linked the goals. Here, to syntactically verify data, the developer has the operator type it twice, which is helpful for data integrity. However, the NFR Assistant system detects that this is bad for user friendliness (the \correlation link," shown as a dashed line, is negative), which results in the user friendliness goal not being met. Correlation links (dashed lines) propagate values in the same way that evaluation links to.
Selecting Di erent Implementation Alternatives. Recognizing that system friendliness is important for users, the developer may consider ways of achieving this goal, such as implementation alternatives presented in Figure 5 (an extension of the lower left part in Figure 4). These include another user-oriented method { menu-driven input, and as system-provided checking { a dictionary of used terms, and using n-grams, which supports automatic recognition of misspelled words. In the example, n-grams are selected, so that syntactic checking remains accomplished, albeit by a di erent method, which contributes positively to user friendliness. In addition, the chosen methods for displaying all attributes early and relevant attributes later remain unchanged
Building Quality into Case-Based Reasoning Systems
377
from Figure 4, hence continue to contribute positively to user friendliness, which is now accomplished. User Friendliness [System]
Syntactic Check [Data]
++
-
+ +
+ Use Later [Relevant Attributes]
BySystem
ByUser
--
Claim["User Friendliness Important"]
Use Early [All Attributes] Informativeness [System]
Robustness [Presentation]
n-Grams Dictionary Menu RetypeData
Fig. 5. A Re-examination of Methods for Syntactic Checking. This is another example of dealing with change { namely, a change in implementation alternatives. The net result is that the developer's expertise was used to accomplish the remaining top goal of user friendliness, while maintaining robustness and informativeness. This was done by reusing the information already captured in Figure 4, which dealt with several NFRs.
4 Conclusions We are concerned with quality issues in decision support for complex information systems. We have presented an approach, called QualityCBR, for dealing with non-functional requirements for case-based reasoning systems. This integrates the NFR Framework's systematic process for building quality with the TA3 CBR system, intended for decision support. In developing QualityCBR, catalogues have been organized to represent diverse knowledge concerning CBR, NFRs, IVF, and development techniques. By drawing on several axes (e.g., CBR and performance), we can focus on small groups of speci c methods. This approach is similar to the organization of information system performance requirements issues [34]. We feel that the use of such catalogues is helpful in dealing with NFRs in medical computing and other complex domains, public and private. To demonstrate how a developer can use QualityCBR to deal with con icting and changing requirements, we illustrated its use in a medical domain. A variety of NFRs (e.g., performance, security, informativeness), and tradeo s between individual requirements have been considered. We also found that the NFR Framework's development graphs and change facilities [7] made the process of dealing with change easier. In this paper we have considered changes in priorities of NFRs and in implementation techniques. This is consistent with results of using the NFR Framework to deal with changes in requirements for a commercial system.
378
I. Jurisica and B.A. Nixon
TA3's performance evaluation has been conducted on several domains: prediction and knowledge mining in medicine [24], [24], control task in robotic domains [21], character recognition [22], iterative browsing and intelligent retrieval [20]. Each domain has di erent characteristics; this helps evaluation of di erent aspects of the system. We have evaluated both the competence [24] and scalability [23] of the system. It would be interesting to see if QualityCBR could be used to use other goaloriented approaches to requirements engineering, e.g., [8, 11, 12]. This would draw on several facilities, such as representation of goals, priorities, and positive and negative links. We would like to conduct fuller studies of applying TA3 to a variety of areas, both public and private, such as medicine, engineering and commerce, which require a variety of NFRs. Notably, we plan to explore the capability of using QualityCBR during building engineering applications, such as robotics [21], where real time response is critical. For example, the use of an \any time system" (which must produce a valid answer at any time) entails exible and adaptive procedures to meet accuracy and safety requirements [19]. These steps will help us to better asses the generality of the approach and proposed combined tools to evaluate its costs and bene ts. Studies should use a methodology, such as [27] which allows us to have the kind of con dence in the results that one would have in using the scienti c method. An important direction for future work is to apply CBR to the NFR Framework and its associated tool. For example, sets of development graphs for a variety of systems could be examined and analyzed to nd patterns (templates) of sequences of method applications. This could be aided by facilities for critiquing and rationalizing speci cations [11]. Such templates could then be used as larger building blocks when using the NFR Framework to develop a variety of systems. Thus, CBR would provide underlying technology for a reuse assistant for the NFR Framework. We trust that building quality into CBR, and using CBR in tools for dealing with NFRs, will aid the development of complex informationsystems for a variety of public and private domains.2
References 1. R. Agrawal, T. Imielinski, and A. Swami. Database mining: A performance perspective. IEEE Transactions on Knowledge and Data Eng. Learning and Discovery 2 Acknowledgments. The authors were with the Dept. of Computer Science, University of Toronto, when this paper was initially prepared. This research was supported by the Information Technology Research Centre of Ontario, Canadian Consortium for Software Engineering Research and the IBM Centre for Advanced Studies. We thank Robert F. Casper and Andrea Jurisicova for much helpful information on IVF procedures. Over the years we have bene ted from the insight and direction of Professors John Mylopoulos and Janice Glasgow. This paper bene ts from earlier NFR work with Lawrence Chung and Eric Yu. Our families have been a constant source of support.
Building Quality into Case-Based Reasoning Systems
379
in Knowledge-Based Databases, 5(6):914{925, 1993. 2. R. Behi and M. Nolan. Ethical issues in research. British Journal of Nursing, 4(12):712{716, 1995. 3. L. Chung. Dealing with security requirements during the development of information systems. In Proc. 5th Int. Conf. on Advanced Information Systems Engineering, pages 234{251, Paris, France, 1993. Springer-Verlag. 4. L. Chung. Representing and Using Non-Functional Requirements: A ProcessOriented Approach. PhD thesis, Dept. of Computer Science, Univ. of Toronto, 1993. 5. L. Chung and B. A. Nixon. Dealing with Non-Functional Requirements: Three Experimental Studies of a Process-Oriented Approach. In Proc. 17th Int. Conf. on Software Eng., pages 25{37, Seattle, WA, 1995. 6. L. Chung, B. A. Nixon, J. Mylopoulos, and E. Yu. Non-Functional Requirements in Software Engineering. In preparation, 1998. 7. L. Chung, B. A. Nixon, and E. Yu. Dealing with Change: An Approach using Non-Functional Requirements. Requirements Engineering, 1(4):238{260, 1996. An earlier version, Using Non-Functional Requirements to Systematically Support Change, appeared in Proc. 2nd IEEE Int. Symp. on Requirements Eng., York, U.K., 1995, pp. 132-139. 8. A. Dardenne, A. van Lamsweerde, and S. Fickas. Goal-directed requirements acquisition. Science of Computer Programming, 20:3{50, 1993. 9. D. E. Denning, P. J. Denning, and M. D. Schwartz. The tracker: A threat to statistical database security. ACM TODS, 4:76{96, 1979. 10. C.L. Early and L. C. Strong. Certi cates of con dentiality: A valuable tool for protecting genetic data. American Journal of Human Genetics, 57(3):727{731, 1995. 11. S. Fickas and P. Nagarajan. Being suspicious: Critiquing problem speci cations. In Proc. AAAI{88, pages 19{24, Saint Paul, MN, 1988. 12. S. F. Fickas. Automating the transformational development of software. IEEE Trans. on Software Eng., SE{11(11):1268{1277, 1985. 13. F.H. France and P.N. Gaunt. The need for security { A clinical view. International Journal of Bio-Medical Computing, 35(Suppl. 1):189{194, 1994. 14. S. Furnell, P. Gaunt, G. Pangalos, P. Sanders, and M. Warren. A generic methodology for health care data security. Medical Informatics, 19(3):229{245, 1994. 15. S. Greenspan, J. Mylopoulos, and A. Borgida. On Formal Requirements Modeling Languages: RML Revisited. In Proceedings, 16th International Conference on Software Engineering, pages 135{147, Sorrento, Italy, 1994. 16. F. Honig. When you can't ask their name: Linking anonymous respondents with the Hogben number. Australian Journal of Public Health, 19(1):94{96, 1995. 17. W. F. Hyslop. Performance Prediction of Relational Database Management Systems. PhD thesis, Dept. of Computer Science, Univ. of Toronto, 1991. 18. M. Jarke. ConceptBase V3.1 User Manual. Univ. of Passau, 1992. 19. I. Jurisica. Supporting exibility. A case-based reasoning approach. In The AAAI Fall Symposium. Flexible Computation in Intelligent Systems: Results, Issues, and Opportunities, Cambridge, MA, 1996. 20. I. Jurisica. Similarity-based retrieval for diverse Bookshelf software repository users. In IBM CASCON Conference, pages 224{235, Toronto, Canada, 1997. 21. I. Jurisica and J. Glasgow. A case-based reasoning approach to learning control. In 5th International Conference on Data and Knowledge Systems for Manufacturing and Engineering, DKSME-96, Phoenix, AZ, 1996.
380
I. Jurisica and B.A. Nixon
22. I. Jurisica and J. Glasgow. Case-based classi cation using similarity-based retrieval. International Journal of Arti cial Intelligence Tools. Special Issue of IEEE ICTAI-96 Best Papers, 6(4):511{536, 1997. 23. I. Jurisica and J. Glasgow. An ecient approach to iterative browsing and retrieval for case-based reasoning. In Angel Pasqual del Pobil, Jose Mira, and Moonis Ali, editors, Lecture Notes in Computer Science, IEA/AIE*98. Springer-Verlag, 1998. 24. I. Jurisica, J. Mylopoulos, J. Glasgow, H. Shapiro, and R. F. Casper. Case-based reasoning in IVF: Prediction and knowledge mining. Arti cial Intelligence in Medicine, 12(1):1{24, 1998. 25. D. Leake, editor. Case-Based Reasoning: Experiences, lessons and future directions. AAAI Press, 1996. 26. D. Leake, A. Kinley, and D. Wilson. Case-based similarity assessment: Estimating adaptability from experience. In Proc. of the AAAI-97, 1997. 27. A. S. Lee. A scienti c methodology for MIS case studies. MIS Quarterly, pages 30{50, 1991. 28. J. Lee. Extending the Potts and Bruns Model for Recording Design Rationale. In Proc., 13th Int. Conf. on Software Eng., pages 114{125, Austin, Texas, 1991. 29. P. M. McShane. Customized comparative clinical results assessment of your IVF program. IVF America, 1993. 30. J. Mylopoulos, A. Borgida, M. Jarke, and M. Koubarakis. Telos: Representing knowledge about information systems. ACM Transactions on Information Systems, 8(4):325{362, 1990. 31. J. Mylopoulos, L. Chung, and B. Nixon. Representing and Using Non-Functional Requirements: A Process-Oriented Approach. IEEE Transactions on Software Engineering, 18:483{497, 1992. 32. H.B. Newcombe. When privacy threatens public health. Canadian Journal of Public Health. Revue Canadienne de Sante Publique., 83(3):188{192, 1995. 33. B. A. Nixon. Dealing with performance requirements during the development of information systems. In Proc. IEEE International Symposium on Requirements Engineering, pages 42{49, San Diego, CA, 1994. 34. B. A. Nixon. Representing and using performance requirements during the development of information systems. In Proc. 4th Int. Conf. on Extending Database Technology, pages 187{200, Cambridge, U.K., 1994. 35. B. A. Nixon. Performance Requirements for Information Systems. PhD thesis, Dept. of Computer Science, Univ. of Toronto, 1997. 36. B. A. Nixon, K. L. Chung, D. Lauzon, A. Borgida, J. Mylopoulos, and M. Stanley. Design of a compiler for a semantic data model. In J. W. Schmidt and C. Thanos, editors, Foundations of Knowledge Base Management, pages 293{343. SpringerVerlag, 1989. 37. J. Ortega. On the informativeness of the DNA promoter sequences domain theory. Journal of Arti cial Intelligence Research, 2:361{367, 1995. Research Note. 38. H. A. Simon. The Sciences of the Arti cial, 2nd Edition. MIT Press, Cambridge, MA, 1981. 39. C. U. Smith. Performance Engineering of Software Systems. Addison-Wesley, Reading, MA, 1990. 40. P. R. Thagard, K. J. Holyoak, G. Nelson, and D. Gotchfeld. Analog retrieval by constraint satisfaction. Arti cial Intelligence, 46:259{310, 1990.
Assembly Techniques for Method Engineering Sjaak Brinkkemper1, Motoshi Saeki2, Frank Harmsen3 1
Baan Company R & D, P.O. Box 143, 3770 AC Barneveld, the Netherlands, [email protected] 2 Tokyo Institute of Technology, Ookayama 2-12-1, Meguro-ku, Tokyo 152, Japan, [email protected] 3 Moret Ernst & Young, P.O. Box 3101, 3502 GC Utrecht, the Netherlands, [email protected]
Abstract. As projects for developing information systems are getting larger and more complicated, we need to have more advanced development methods suitable for every development situation. Method engineering is the discipline to construct new methods from parts of existing methods, called method fragments. To achieve this objective, we need to clarify how to model the existing methods and how to assemble method fragments into new project-specific methods, so-called situational methods. Especially, to produce meaningful methods, we should impose some constraints or rules on method assembly processes. In this paper, we propose a framework for hierarchical method modelling (meta-modelling) from three orthogonal dimensions: perspectives, abstraction and granularity. According to each dimension, methods and/or method fragments are hierarchically modelled and classified. Furthermore, we present a method assembly mechanism and its formalization as a set of rules. These rules are presented in first order predicate logic and play an important role in the assembly process of meaningful methods from existing method fragments. The benefit of our technique is illustrated by an example of method assembly, namely the integration of the Object Model and Harel's Statechart into Objectcharts.
1
Introduction
The size and complexity of projects for developing information systems are becoming larger and more complicated. Therefore, development methods and supporting tools turn one of the most significant key factors to achieve great success of development projects. Until now, many methods such as structured analysis/design [De Marco 78] and object-oriented analysis/design [Rumbaugh91] have been proposed and many textbooks have been published. The information-technology industry is putting the existing methods and corresponding supporting tools into practice in real development projects. However, much time and effort is spent on applying the methods effectively in these projects. One of the reasons is that contemporary methods are too general and includes some parts, which do not fit to the characteristics of real projects and their contexts. To enhance the effect of methods, for each of real projects, we need to adapt the methods or construct the new ones so that they can fit to the project. Method Engineering, in particular Situational Method Engineering [Harmsen 94, Brinkkemper 96] is the discipline to build project-specific methods, called situational methods, from parts of the existing methods, called method fragments. This technique is coined method assembly. In fact, many methods can be considered to be the result B. Pernici and C. Thanos (Eds.): CAiSE’98, LNCS 1413, pp. 381-400, 1998. Springer-Verlag Berlin Heidelberg 1998
382
S. Brinkkemper, M. Saeki, and F. Harmsen
of applying method assembly. For instance, OMT [Rumbaugh 91] has been built from the existing fragments Object Class Diagram (extended Entity Relationship Diagram), State Transition Diagram, Message Sequence Chart and Data Flow Diagram, all originating from other method sources. This example shows that method assembly produced a powerful new method that could model complicated systems from multiple viewpoints: object view, behavioural view and functional view. Therefore, method assembly is a significant technique to construct both situational methods and powerful methods with multiple viewpoints. To assemble method fragments into a meaningful method, we need a procedure and representation to model method fragments and impose some constraints or rules on method assembly processes. If we allow assembly arbitrary method fragments, we may get a meaningless method. For example, it makes no sense to assemble Entity Relationship Diagram and Object Class Diagram in the same level of abstraction. Thus, the modelling technique for method fragments, so called meta-modelling technique should be able to include the formalization of this kind of constraints or rules to avoid producing meaningless methods. Several researchers applied very adequate meta-modelling techniques based on Entity Relationship Model [Brinkkemper 91, Sorenson 88, Nuseibeh 95], Attribute Grammars [Katayama 89, Song 94], Predicate Logic [Brinkkemper 91, Saeki 94, Nuseibeh 95] and Quark Model [Ajisaka 96] for various method engineering purposes (see section 6). Some of these works discuss the inconsistency of products when we assemble several methods into one, however, none of them referred to method assembly function itself yet. Song investigated existing methods, such as OMT and Ward/Mellor’s Real Time SDM [Ward 85], and classified the way various methods are put together [Song 95]. Guidelines or rules to assemble methods were not elaborated in this study. Furthermore, as discussed later in section 6, his classification is fully included in ours. In this paper, we propose a framework for hierarchical meta-modelling from three orthogonal dimensions: perspective, abstraction and granularity. According to each dimension, methods and method fragments are hierarchically modelled and classified. According to this classification of method fragments, we can provide the guideline for meaningful method assembly. That is to say, we can suggest that method fragments, which belong to a specific class can be meaningfully assembled. For example, we can sufficiently construct a meaningful method from method fragments with the same granularity level. In another example, it is not preferable to assemble the method fragments belonging to the same specific category such as Entity Relationship Diagram and Object Class Diagram, as the latter can be seen as an extension of the former. These kinds of guideline and constraints can be formalized as a set of rules based on our multiple hierarchical dimensions. These rules can be presented in first order predicate logic and play an important role on clarifying method assembly mechanism. This paper is organised as follows. In the next section, we begin with illustrating a simple example of the method fragment Statechart and introduce three orthogonal dimensions for classification of method fragments. Section 3 presents method assembly by using example of assembling Object Model and Statechart into the new
Assembly Techniques for Method Engineering
383
method fragment Objectchart. This example suggests to us what kind of guidelines or constraints are required to method assembly. We discuss these guidelines and constraints, and their formalization in section 4. Sections 5 and 6 summarize related work and our work respectively.
2 2.1
A Classification Framework For Method Fragments Method Fragments
We begin with an example of the description of the method fragment of Harel's Statechart. Statecharts can be seen an extension of finite state transition diagram to specify reactive systems [Harel 90]. To avoid the explosion of the number of states occurring when we specify complicated systems with usual state transition machines, it adopted two types of structuring techniques for states, i.e. hierarchically decomposition of states: one is called AND decomposition for concurrency, and the other one is OR decomposition for state-clustering. The description of the method fragment is illustrated in the meta-model in Fig. 1 in the notation of Entity Relationship Attribute Diagrams. (To avoid confusion, we use the terms concept, association and property in method fragments instead of entity, relationship and attribute.) The Statechart technique comprises four concepts: State, Transition, Event and Firing condition. If a firing condition associated with a transition holds, the transition can occur and the system can change a state (called source state) to a destination state. During transition, the system can output or send an event to the other Statecharts. Firing conditions can be specified with predicates and/or receipt of these events. So we can have four associations among the three concepts, and two associations on the state concept for expressing AND decomposition and OR decomposition. Note that the meta-model does not include representational information, e.g. a state is represented in a rounded box in a diagram, and events are denoted by arrows. We define this kind of information as another aspect of method modelling and discuss it in the next section. AND-decomposition has
Event
is source of
State
Transition is destination of
has
OR-decomposition
Fig.1 Statechart Method Fragment
Firing Condition
384
S. Brinkkemper, M. Saeki, and F. Harmsen
2.2
Classification of Method Fragments
Method fragments are classified according to the dimensions perspective, abstraction level, and layer of granularity. First, the perspective dimension of the classification considers the product perspective and the process perspective on methods. Product fragments represent deliverables, milestone documents, models, diagrams, etc. Process fragments represent the stages, activities and tasks to be carried out. Fig.1 is a description of the product perspective. The abstraction dimension constitutes of the conceptual level and the technical level. Method fragments on the conceptual level are descriptions of information systems development methods or part thereof. Technical method fragments are implementable specifications of the operational parts of a method, i.e. the tools. Some conceptual fragments are to be supported by tools, and must therefore be accompanied by corresponding technical fragments. One conceptual method fragment can be related to several external and technical method fragments. The conceptual method fragment is shown in Fig. 1, whereas the corresponding technical fragment is the STATEMATE tool for specifying Statecharts [Harel 90]. One of the most important and main discriminating properties of method fragments is the granularity layer at which they reside. Such a layer can be compared with a decomposition level in a method. A method, from the process perspective, usually consists of stages, which are further partitioned into activities and individual steps. A similar decomposition can be made of product fragments, with the entire system at the top of the tree, which is subsequently decomposed into milestone deliverables, model, model components, and concepts. Research into several applications of method engineering [Brinkkemper 96] shows that methods can be projected on this classification. A method fragment can reside on one of five possible granularity layers: • Method, which addresses the complete method for developing the information system. For instance, the Information Engineering method resides on this granularity layer. • Stage, which addresses a segment of the life-cycle of the information system. An example of a method fragment residing on the Stage layer is a Technical Design Report. Another example of a Stage method fragment is a CASE tool supporting Information Engineering s Business Area Analysis [Martin 90] stage.
• Model, which addresses a perspective [Olle 91] of the information system. Such a perspective is an aspect system of an abstraction level. Examples of method fragments residing on this layer are the Data Model, and the User Interface Model. • Diagram, addressing the representation of a view of a Model layer method fragment. For instance, the Object Diagram and the Class Hierarchy both address the data perspective, but in another representation. The Statechart resides on this granularity layer, as well as the modelling procedure to produce it.
Assembly Techniques for Method Engineering
385
• Concept, which addresses the concepts and associations of the method fragments on the Diagram layer, as well as the manipulations defined on them. Concepts are subsystems of Diagram layer method fragments. Examples are: Entity, Entity is involved in Relationship, and Identify entities
3 3.1
Method Assembly Technique Method Assembly in the Product Perspective
In this section, we introduce a simple example of method assembly — assembling Object Model in Object-Oriented Analysis/Design and Statechart to Objectchart. Objectchart, proposed in [Coleman 91], is an extension of Statechart to model reactive systems from an object-oriented view. Our framework of method assembly can explain how Objectchart was composed from the existing method fragments Object Model and Statechart. The Object Model specifies a system as a set of objects communicating with each other. Objects have their specific attributes and change their values through interobject communication. By sending messages to the other objects (or itself) an object requires of them (or itself) to provide the service that they (or it) encapsulatedly have. The objects that are requested perform their service and may change their attribute values and/or return the computed results. Objects having the same attributes and services are modelled with a Class, which is a kind of template. Fig. 2 shows the method fragment description of the Object Model at Diagram layer from conceptual level and product perspective.
2EMHFW KDV &ODVV KDV $WWULEXWH
KDV
6HUYLFH
SDUWLFLSDWHV LQ $VVRFLDWLRQ
Fig.2 Object Model Method Fragment
386
S. Brinkkemper, M. Saeki, and F. Harmsen
Suppose now we have to produce Objectchart by assembling these two method fragments i.e. the method models of Figs. 1 and 2. Fig. 3 shows the resulting method fragment of Objectchart in the same level, perspective and layer. As for this assembly process, we should note that the two method fragments belong to the same category in our three dimensional classification: conceptual level in abstraction, Diagram layer in granularity, and product in perspective. In addition we have product perspective of Objectchart in conceptual level and in Diagram Layer. Thus the method fragments with the same category can be assembled and we can get a new method with the same category.
2EMHFW 0RGHO
6WDWHFKDUW (YHQW
6HUYLFH
FRQVLVWV RI
2EMHFW KDV
EHORQJV WR
&ODVV
KDV
UHIHUV WR
$VVRFLDWLRQ
SDUWLFLSDWHV LQ KDV
$WWULEXWH
KDV
UHIHUV WR LV VRXUFH RI
7UDQVLWLRQ
6WDWH
LV DQQRWDWHG ZLWK
LVBKLGGHQ"
UHIHUV WR
LV GHVWLQDWLRQ RI KDV
KDV
3RVW FRQGLWLRQ )LULQJ &RQGLWLRQ
Fig. 3
Objectchart : Method Assembly in the Product Perspective
The Statechart and Object Model are amalgamated to Objectchart by the following constructions: 1) A Class has a Statechart, which specifies its behaviour. 2) Attributes of a Class may be annotated to States in its Statechart. This indicates which attribute values are meaningful or visible in a specific state. 3) An Event issued during a Transition is a request of a Service to the other Object.
Assembly Techniques for Method Engineering
4) A Transition may change an Attribute value of an Object.
387
The first three constructions allow us to introduce new associations has between Class and State, is annotated with between Attribute and State, and consists of . The concept Object participating in consist of stands for the object of which a service is required, i.e. a receiver of the event. Furthermore, we employ the new concept “Post condition” for specifying the change of attribute value when a transition occurs. Therefore, post conditions can define the effect of service-execution on attributes.
Let's explore what manipulations were made and what kinds of constraints could be considered in this example. The basic manipulations that we applied here are: 1) Addition of a new concept (Post condition), 2) Addition of a new association (is_annotated_with, consists_of, has), 3) Addition of a new property (is_hidden). First of all, when we assemble two method fragments, we should introduce at least one new concept or association. If we did not introduce anything, it would mean that a method fragment was completely included in another one. This case might be meaningless because we could not find the effect of this method assembly and the result was the same as the containing method fragment. This applies for the meaningless example of assembling ERD and Object Class Diagram (the super class of ERD), which we mentioned in section 1. Furthermore, at least one connection between the two method fragments through newly introduced associations and/or concepts should be introduced, because the two method fragments are to be conceptually connected by the method assembly. Consequently, these constraints can be generalized as Rule 1)
At least one concept, association or property should be newly introduced to each method fragment to be assembled, i.e. a method fragment to be assembled should not be a subset of another.
Rule 2)
We should have at least one concept and/or association that connects between two method fragments to be assembled.
Rule 3)
If we add new concepts, they should be connectors to both of the assembled method fragments.
Rule 4)
If we add new associations, the two method fragments to be assembled should participate in them.
The following additional rules can easily be determined, whose explanation we omit. Rule 5)
There are no isolated parts in the resulting method fragments.
Rule 6)
There are no concepts which have the same name and which have the different occurrences in a method description.
These rules apply for method fragments in the conceptual level and diagram layer. If the method fragment to be assembled is related to the other levels or layers, the effect
388
S. Brinkkemper, M. Saeki, and F. Harmsen
of assembly propagates to the others. It means that we should have the other types of rules. For example, the different concepts on the conceptual level should have different representation forms (notation) on the technical level. We will discuss a more elaborated style of rules and their formalization in section 4.
3.2
Method Assembly in the Process Perspective
In the previous example, we illustrated product-perspective method assembly. Next, we turn to discuss the process-perspective method assembly also with the help of an example. Suppose we have the process descriptions for Object Model and for Statechart in Diagram layer at our disposal, e.g. for Object Model:
Draw an Object Model O1) Identify objects and classes, O2) Identify relationships, O3) Identify attributes and services. and for Statechart:
Draw a Statechart S1) Identify states, S2) Identify state changes and their triggers, S3) Cluster states, and so on. According to [Coleman 92], the recommended procedure for modelling Objectcharts is as follows:
Draw an Objectchart OC1) Draw an Object Model, OC2) For each significant class, Draw a Statechart, and OC3) Refine the Statechart to an Objectchart by adding post conditions and annotating states of the Statechart with attributes. This procedure is constructed from the two process method fragments, Object Model (step OC1)) and Statechart (step OC2)) and seems to be natural. In more detail, between steps OC1) and OC2), we find that we should perform the activity of identifying the relationship has between Class and State shown in the Fig. 3. The concept “Post condition” and its associations, say refers to , and the association is annotated with are identified while the step OC3) is being performed. It means that newly added concepts and associations to connect the product-perspective method fragments to be assembled should not be identified until the associated concepts are identified. In fact, it is difficult for us to identify the association has between classes and states before we have identified classes or identified states and we should avoid this execution order of the activities (see also Fig. 4).
Assembly Techniques for Method Engineering
Rule 7)
389
The activity of identifying the added concepts and relationships that are newly introduced for method assembly should be performed after their associated concepts are identified.
The rule mentioned above provides a criterion to make meaningful and useful procedures from manipulations on concepts and associations in Diagram Layer. Similarly, we can easily have the rule : we should not identify any associations until we identify their associated concepts in Diagram Layer. So the first step of method procedure should be identifying some concepts. This results from the natural execution order of human perception.
O1: Identify Objects and Classes
S1: Identify States input
List of Objects and Classes
List of States
S2: Identify State changes and Triggers
O2: Identify Associations
Diagram with Classes and Associations
State Transition Diagram input
O3: Identify Attributes and Services
S3: Clustering States ...
Object Model Diagram
Statechart
OC1: Draw an Object Model (A)
OC2: Draw a Statechart (B) OC3: Refine Statecharts
Objectchart
Draw an Objectchart (C)
Fig. 4 Method Assembly in the Process Perspective
Another type of rules relates to the input/output order of products to activities. For example, the activity step O2) in Object Model consumes the identified objects and classes as its inputs which are produced by the step O1). The point in method assembly processes is what input-output relationships are added and/or changed. In this example, as shown in Fig. 4, the step OC2) in Objectchart, which resulted from steps S1), S2) and S3) in Statechart, should consume the identified classes as its
390
S. Brinkkemper, M. Saeki, and F. Harmsen
inputs. They are the output of the step O1) in Object Model, i.e. another method fragment. Therefore we can have the following rule: Rule 8) Let A and B be the two method fragments to be assembled, and C the new method fragment. In C, we should have at least one product which is the output of A and which is the input of B, or the other way round. This rule means that either of the method fragments to be assembled, say A, should produce input to the activities of B in the new method C. More examples of method assembly rules in process perspective will be shown in section 4.
3.3
Discussion of Method Assembly on Three Dimensions
As we have shown in section 2, method fragments can be considered on three dimensions: perspective, abstraction level and granularity layer. These dimensions can be used to improve, speed up, and simplify the method assembly process. We illustrate this with the following example. Assembling Object Model and Statechart, which are product fragments at the Diagram layer and at the conceptual level, implies the assembly of method fragments addressing the other perspective, abstraction level, and granularity layers. Associated with the Statechart and Object Model product fragments are modeling procedures, i.e. process fragments. The assembled modeling procedure results from the components of each of these two process fragments. Some of the rules that apply are: Rule 9) Each product fragment should be produced by a “corresponding” process fragment. Rule 10) Suppose a product fragment has been assembled. The process fragment that produces this product fragment consists of the process fragments that produce the components of the product fragment. Also associated with the conceptual method fragments mentioned above are technical method fragments, such as Object Model and Statechart diagram editors, a repository to store object models and Statecharts, and a process manager to support the modeling procedures for object models and Statecharts. Similarly, the assembly of these technical method fragments results from the assembly of the corresponding conceptual method fragments: Rule 11)
A technical method fragment should supports a conceptual method fragment.
The assembly of fragments at the Diagram layer has also implications for the components of these fragments, which are at the Concept layer. In general, assembly of two method fragments results in the assembly of method fragments of lower granularity layers. As we have seen in section 3.1, the assembly of Object Model and Statechart results in the assembly of Service and Event, Class and State, and Attribute and Firing Condition. A rule that applies to this is: Rule 12)
If an association exists between two product fragments, there should exist at least one association between their respective components
Assembly Techniques for Method Engineering
391
We have taken in the above example the assembly of conceptual product fragments at the Diagram layer as a starting point. However, the starting point can be at any combination of perspective, abstraction level, and granularity layer. Obviously, whatever starting point is used, the result of one assembly action is a cascade of other actions within the three-dimensional framework.
4 4.1
Method Assembly : Guideline and Formalization Requirements for Method Assembly
Method assembly should ensure that the selected method fragments are mutually adjusted, i.e. they have to be combined in such a way that the resulting situational method does not contain any defects or inconsistencies. Several types of defects can appear: • Internal incompleteness, which is the case if a method fragment requires another method fragment that is not present in the situational method. For instance, a data model has been selected without the corresponding modelling procedure and tool. • Inconsistency, which is the case if the selection of a method fragment contradicts the selection of another method fragment. For instance, two similar data modelling techniques have been selected without any additional reason. • Inapplicability, which is the case if method fragments cannot be applied by project members, due to insufficient capability. All these issues relate to the internal or situation-independent quality [Hoef 95] of a situational method, i.e. the quality of a method without taking into consideration the situation in which the method is applied. The two most important criteria are: • Completeness: the situational method contains all the method fragments that are referred to by other fragments in the situational method. • Consistency: all activities, products, tools and people plus their -mutualrelationships in a situational method do not contain any contradiction and are thus mutually consistent. Furthermore, we distinguish the following method internal quality criteria that are not treated in this paper for the sake of brevity and their details is in [Harmsen 97]: • Efficiency: the method can be performed at minimal cost and effort • Reliability: the method is semantically correct and meaningful • Applicability: the developers are able to apply the situational method The effort to achieve situation-independent quality of method fragments is considerable. Method fragments can be combined in a lot of ways, many of which are meaningless. Moreover, method fragments require other method fragments to be meaningful in a situational method, or require certain skills from the actors related to them. This is illustrated by the following small example. Suppose a process
392
S. Brinkkemper, M. Saeki, and F. Harmsen
perspective method fragment Draw an Object Model (shown in sect. 3.2) has been selected. The following should be at least verified ; 1) No similar method fragment already exists in the situational method, 2) The specification of the Object Model produced by the process fragment is selected, 3) Actors have the expertise to deal with this process fragment, and 4) The products required are produced by preceding selected process fragments (See also the examples in sect. 3.1 and sect. 3.2). Internal method quality can only be achieved by a set of guidelines on the Method Engineering level. These formalized guidelines are presented in the form of axioms, which can be considered an extension of the set of axioms, corollaries and theorems presented in section 4. The axioms are grouped by the various quality criteria.
4.2
Classification of Method Assembly
In this section, the general internal quality requirements completeness and consistency are further partitioned by means of the three-dimensional classification framework. Completeness is partitioned into: •
Input/output completeness, stating that if a process fragment requiring or manipulating a product fragment is selected, then that product fragment should be available in the situational method. Input/output completeness applies to the interaction of the two perspectives.
•
Content completeness, stating that if a method fragment is selected, all of its contents have to be available too. Contents completeness applies to the relationship between granularity layers.
•
Process completeness, requiring that all product fragments have to be, in some way, produced. Process completeness is related to the interaction of the two perspectives.
•
Association completeness, requiring that product fragments on certain layers are always involved in an association, and that associations always involve product fragments. Association completeness relates to the product perspective.
•
Support completeness, requiring that technical method fragments support conceptual method fragments. Support completeness applies to the relationship between abstraction levels. Consistency is partitioned into:
•
Precedence consistency, requiring that product fragments and process fragments are placed in the right order in the situational method. This type of consistency applies to the interaction between perspectives.
•
Perspective consistency, requiring that the contents of product fragments is consistent with the contents of process fragments. Perspective consistency also
Assembly Techniques for Method Engineering
393
applies to the interaction between perspectives. •
Support consistency, requiring that technical method fragments are mutually consistent. Support consistency relates to the relationships of technical method fragments.
•
Granularity consistency, which imposes that the granularity layers of related method fragments are similar, and that their contents are mutually consistent. This type of consistency applies to the interaction between granularity layers.
•
Concurrence consistency, which requires parallel activities to be properly synchronized. Concurrence consistency relates to the interaction of process fragments.
Note that our concepts of “completeness” and “consistency” are syntactical constraints on descriptions of method fragments written in Entity Relationship Model. To formalize actual method assembly processes more rigorously and precisely, we should consider some aspects of the meaning of method fragments. In the example of Objectchart, we associated the concept “Attribute” with “State”. The question is in whatever method assembly we can always do it. The answer depends on the semantics of these concepts in the method fragments. How to specify the semantics of method fragments for method assembly is one of the most important and interesting future topics. In the next sub-section, each of these categories will be elaborated by means of an example taken from the Objectchart case.
4.3
Method Assembly Rules
4.3.1 Some Definitions As noticed before, the natural language representation of method assembly rules creates some problems regarding ambiguity and implementability. Therefore we have formalized our theory regarding method fragments, and expressed the rules in that formalization. In this sub-section, we only show the part of the formalization required in the context of this paper. Moreover, we give examples of rules, some of which are formalized well. The formalization employs the following notions: • Set, which represents a category of similar method fragments. • Predicate, which represents a relationship between Method Base concepts. • Function, which represents the assignment of the method fragment properties to method fragments • The usual logical quantifiers and operators. • The operators <, =, ∈, ⊂, ∪ and ∩. The following sets are defined:
394
S. Brinkkemper, M. Saeki, and F. Harmsen
M = C ∪ T, the set of method fragments C = R ∪ P, the set of conceptual method fragments: e.g. Draw an Object Model, Object Model, Statechart, Identify Classes and Objects, Class, Object, Service, Transition has Event, List of States.
R the set of product fragments, e.g. Object Model, Statechart, List of States P the set of process fragments, e.g. Class, Object, Service, State, Event. CN ⊆ R , the set of concepts, e.g. Class, Object, Service, State, Event.of concepts are postulated A ⊆ R, the set of associations, e.g. Transition with Attribute.
has Event, State is annotated
T the set of technical method fragments. If a method fragment is selected for inclusion in a situational method, it is indexed with an s , for instance: Rs is the set of selected product fragments.
The following predicates are used in this section:
q
q
• contents and contents* ⊆ R R ∪ P P to represent the non-transitive and transitive consists-of relationship between method fragments, e.g. contents(Class, Object Model);
q
• manipulation ⊆ P R, to represent the fact that a process fragment manipulates (i.e. produces, updates, etc.) a certain product fragment, e.g. manipulation(Draw an Objectchart, Objectchart);
q
• involvement ⊆ A R, to represent the fact that an association involves a product fragment , e.g. involvement (is annotated with, Objectchart);
q
• prerequisite ⊆ P R, to represent the fact that a process fragment requires a product fragment for its execution, e.g. prerequisite(Identify Associations, List of Classes and Objects);
q
• precedence ⊆ P P, denote the precedence relationship between process fragments, e.g. precedence(Identify Associations, Identify Classes and Objects);
q
• support ⊆ C T, to represent that a technical method fragment supports a conceptual method fragment, e.g. support(Statechart, STATEMATE); • concurrence, to represent the fact that two process fragments can be performed in parallel, e.g. concurrence(Identify Associations(O2), Identify States(S1)) (see Fig.4).
³
• layer: M {Method, Stage, Model, Diagram, Concept},to return the layer of the method fragment (see sect. 2.2), e.g. layer(Objectchart)=Diagram, layer(Class)=Concept.
Assembly Techniques for Method Engineering
395
Below, each type of completeness and consistency, as defined in sect. 4.1, is related to our Objectchart example. We assume that both Object Model, Statechart, and Objectchart should be part of a complete and consistent situational method, Ms 4.3.2 Completeness rules Input/output completeness Step 2 of the Objectchart modeling procedure requires an Object Model. The description of the Object Model should therefore exist in the situational method. In general, the rule is: Required product fragments should have been selected for the method assembly , i.e. ∀p ∈ Ps , r ∈ R [ prerequisite ( p , r ) → r ∈ R ] s
Contents completeness Concepts (product fragments) such as Class, Object, State, Service, Transition etc. should always be part of another product fragment. Note that this is indeed the case, as they are all components of Statechart. In a formalized way, this rule is defined as follows: ∀r1 ∈ R s ∃r2 ∈ R s [ layer ( r1 ) = concept → contents * (r2 , r1 ) ∧ layer (r2 ) ∈ {Model, Diagram}]
Process completeness Suppose the Objectchart is included in the situational method. Then it has to be produced by some process fragment that is also included. In general, selected product fragments at the lowest four granularity layers have to be produced by a selected process fragment, i.e. ∀r ∈ R s ∃p ∈ Ps [layer ( r ) ≠ Concept → manipulation( p, r )]
Association completeness Suppose both the Object Model and State Chart have been selected for inclusion in the situational method. Then they should be connected by at least one association (note, again, that this is the case; they are connected by even more than one association). In general, if more than one diagram layer product fragment has been selected, diagram layer product fragments should be associated with at least one other diagram layer product fragment. (Rule 4)).
396
S. Brinkkemper, M. Saeki, and F. Harmsen
∀r1 , r2 ∈ R s ∃a ∈ As [layer (r1 ) = Diagram ∧ layer ( r2 ) = Diagram ∧ r1 ≠ r2 → involvement ( a, r1 ) ∧ involvement (a, r2 )] Also Rule 3) is an example of an association completeness rule: ∀r1 , r2 ∈ Rs ∃a1 , a 2 ∈ As ∃c ∈ CN s [( layer (r1 ) = Diagram ∧ layer ( r2 ) = Diagram ∧ r1 ≠ r2 ) → involvement ( a1 , r1 ) ∧ involvement (a 2 , r2 ) ∧ involvement (c , r1 ) ∧ involvement (c , r2 )] From these rules we can deduce, that Rule 2) is redundant. Support completeness Suppose the STATEMATE editor was selected for inclusion in our situational method. Then, the Statechart product fragment that is supported by this editor should also be included. In a formalized way, this rule, i.e.Rule 11) is defined as follows: ∀t ∈ Ts , r ∈ R [ support ( r , t ) → r ∈ R ]
4.3.3 Consistency Rule Precedence consistency In the modeling procedure for Objectchart, step OC2 requires an Object Model. This Object Model should be produced by a step before step OC2. In general: a process fragment producing a required product fragment should be placed before the process fragment requiring the product fragment, i.e.
∀p ∈ P , r ∈ R ∃p ∈ P [ prerequisite( p , r ) 1
1 s s 2 s → manipulation( p , r ) ∧ precedence ( p , p )] 2 1 2
This rule is a part of Rule 7). This rule means that we should have at least one new process fragment and this new fragment should not be first in the order of the assembled process fragments.
In the example of Fig. 4, we have a new process fragment Refine Statechart (OC3) , and it cannot be performed before Draw an Objectchart and Draw a Statechart. The above rule specifies the latter part. We can also formalize the former part.
Perspective consistency Objectchart is produced by the modeling procedure presented in section 3.2. The components of Objectchart, its concepts, should be produced by components of this fragment. As a general rule: If a product fragment is produced by a certain process fragment, then all of its contents should be produced by the sub-processes of that process fragment, i.e. ∀p1 , p2 ∈ Ps , r ∈ R s , b ∈ B∃r2 ∈ R s [ manipulation( p1 , r1 ) ∧ contents( p1 , p2 ) → contents( r1 , r2 ) ∧ manipulation( p2 , r2 )]
Assembly Techniques for Method Engineering
397
Granularity consistency An example of a granularity consistency rule is Rule 12) (section 3.4), stating that if two product fragments are associated, there should be at least an association at the Concept layer in their perspective contents as well, i.e.: ∀a1 ∈ As , r1 , r2 ∈ R s , l1 , l 2 ∈ L ∃c1 , c 2 ∈ CN s , a 2 ∈ As [involvement ( a1 , r1 ) ∧ involvement ( a1 , r2 ) → contents * ( r1 , c1 ) ∧ contents * ( r2 , c 2 ) ∧ involvement (a 2 , c1 ) ∧ involvemnet (a 2 , c 2 )]
Concurrence consistency Suppose the Objectchart process fragment consists, to speed up the process, of two steps that are concurrently executed. This may only be the case, if they do not require complete products from each other. So, for instance, steps OC1 and OC2 of the Draw an Objectchart fragment may not be concurrently executed, as step OC2 required some intermediate results produced by step OC1. However, within this fragment some steps can be performed concurrently, e.g. O2 and S1. The concurrence consistency rule is defined as follows: ∀p1 , p 2 ∈ Ps , r ∈ R s [concurrence ( p1 , p 2 ) → ¬( prerequisite( p1 , r ) ∧ manipulation( p 2 , r )) ∧ ¬( prerequisite ( p 2 , r ) ∧ manipulation( p1 , r ))]
5
Related Work
As mentioned before, several meta-modelling techniques were proposed, e.g. they were based on Entity Relationship Model, Attribute Grammar, Predicate Logic and Quark Model. Comparison of meta-modelling techniques and their languages was also discussed in [Harmsen 96]. We pick up a few representatives and discuss their relevance to our work. Almost all approaches to meta-modelling are using Entity Relationship Model (ER). Some applied Predicate Logic to describing the properties, which cannot be represented with just the ER notation. For instance, the Viewpoints approach [Nuseibeh 92] combines ER and Predicate Logic. It aims at constructing a method with multiple views from the existing methods. In other words, we can define the assembly mechanism of the products, which are produced by the different existing methods. The approach also provides the function for defining constraints to maintain consistency on the products that are produced by the existing methods. However, it discusses about the constraints on the assembled products but not constraints on method assembly processes themselves.
398
S. Brinkkemper, M. Saeki, and F. Harmsen
Software Quark Model [Ajisaka 96] tried to formalize a restricted set of atomic concepts, which can specify any kind of software products and it can be considered as a product perspective of meta-modelling. The aim of the model seems to be not method assembly in product level, but maintaining causality relationships among the software products produced in various stages of a software development cycle through atomic concepts. In his article, Song investigated the existing integrated methods, into which several different methods were integrated, and classified method integration from benefitoriented view, i.e. classification criteria is based on what benefit we can get by the integration [Song 95]. He did not use the term ``assembly" but ``integration". According to his classification, we can have two categories: function-driven (a new function is added) and quality-driven (the quality of a method is improved). He also classified these two categories in detail based on which components of methods are integrated, e.g. Artifact Model Integration, Process Integration, Representation Integration and so on. His work is a pioneer of method assembly research. However, he did not discuss how to integrate (assemble) methods or what rules should hold for each category but just classified the existing integration patterns. And, all of his proposed classes are not necessary orthogonal, i.e. an integration is included in several classes. Our framework is completely orthogonal and we have shown some guidelines and rules to produce meaningful methods. Furthermore our classification includes Song's classification. Fig. 3 is an example of Song's Artifact Model Integration, i.e. method assembly in Conceptual Level, Product Perspective and Diagram Layer.
6
Conclusion and Future Work
This paper clarifies how to assemble method fragments into a situational method and formalize rules to construct meaningful methods. We have already extracted over 80 rules thought real method assembly processes. Our rules are general ones which are applicable for arbitrary method assembly, and we may need some rules for specific kinds of method assembly. These rules probably include semantic information on method fragments and on systems to be developed. Our next goal is to assess our generic rules in more complicated and larger-scale assembly processes, e.g. whether our rules are sufficient and minimal to specify method assembly processes as general rules, and to look for specific rules as method assembly knowledge. Our rules are described with predicate logic, so we have a possibility to check method fragments automatically during the assembly processes. To get efficient support, we should consider how our rules can be efficiently executed in our method base system, which stores various kinds of method fragments. As reported elsewhere, we are currently developing the Computer Aided Method Engineering (CAME) tool, called Decamerone [Harmsen 95], which includes a comprehensive method base system. A support function for method assembly processes based on our assembly rules is currently under development. Functionality for adaptive repository generation and customisable process managers is being realised. Next to this, the Method Engineering Language (MEL) is under development [Harmsen 96]. This language allows us to describe method fragments from the various relevant dimensions.
Assembly Techniques for Method Engineering
399
Operators for the manipulation, storage and retrieval of method fragments in the method base have been defined. To clarify which method fragments are suitable and useful for a specific situation is one of the most important research issues and empirical studies are necessary such as [Slooten 96] and [Klooster 97].
References [Ajisaka 96]
Ajisaka,T.. The Software Quark Model: A Universal Model for CASE Repositories. In Journal of Information and Software Technology, 1996.
[Brinkkemper 94] Brinkkemper, S., Method Engineering: Engineering of Information Systems Development Methods and Tools. In Journal of Information and Software Technology, 1996. [Coleman 92]
Coleman,F., Hayes,F. and Bear,S., Introducing Objectcharts or How to Use Statecharts on Object-Oriented Design. IEEE Trans Soft. Eng., Vol.18, No.1, pp.9 -- 18, 1992.
[De Marco 78]
DeMarco, T., Structured Analysis and System Specification, Yourdon Press, 1978.
[Harel 90]
Harel,D., Lachover,H., Naamad,A., Pnueli,A., Politi,M., Sherman,R. Shutull-Trauring,A. and Trakhtenbrot,M., STATEMATE: A Working Environment for the Development of Complex Reactive Systems. IEEE Trans. Soft. Eng., Vol.16, pp.403 -- 414, 1990.
[Harmsen 94]
Harmsen, F., S. Brinkkemper, H. Oei, Situational Method Engineering for Information System Projects. In: Olle, T.W., and A.A. Verrijn Stuart (Eds.), Methods and Associated Tools for the Information Systems Life Cycle, Proceedings of the IFIP WG8.1 Working Conference CRIS 94, North-Holland, pp. 169-194, Amsterdam, 1994.
[Harmsen 95]
Harmsen, F. and S. Brinkkemper, Design and Implementation of a Method Base Management System for a Situational CASE Environment. In: Proceedings of the APSEC 95 Conference, IEEE Computer Society Press, Los Alamitos, CA, 1995.
[Harmsen 96]
Harmsen, F., and M. Saeki, Comparison of Four Method Engineering Languages. In: In: S. Brinkkemper, K. Lyytinen and R. Welke (Eds.), Method Engineering: Principles of Method Construction and Tool Support, Chapman & Hall, pp.209-231, 1996.
[Harmsen 97]
Harmsen, F., Situational Method Engineering. Moret Ernst & Young, 1997
400
S. Brinkkemper, M. Saeki, and F. Harmsen
[Hoef 95]
Hoef, R. van de, and F. Harmsen, Quality Requirements for Situational Methods. In: Grosz, G. (Ed.), In Proceedings of the Sixth Workshop on the Next Generation of CASE Tools, Jyväskylä, Finland, June 1995.
[Katayama 89]
Katayama, T., A Hierarchical and Functional Software Process Description and Its Enaction. In: Proceedings of 11th Int. Conf. on Software Engineering. pp.-343-352, May 1989.
[Klooster 97]
Klooster, M., S. Brinkkemper, F. Harmsen, and G. Wijers, Intranet Facilitated Knowledge Management: A Theory and Tool for Defining Situational Methods. In: A. Olive, J.A. Pastor (Eds.), Proceedings of CAiSE'97. Lecture Notes in Computer Science 1250, Springer Verlag, pp.303-317, 1997.
[Nuseibeh 95]
Nuseibeh, B., J Kramer and A. Finkelstein, Expressing the Relationship between Multiple View in Requirements Specification. In: Proceedings of 15th Int. Conf. on Software Engineering, Baltimore, IEEE Computer Society Press, pp. 187197, 1993.
[Olle 91]
Olle, T.W., J. Hagelstein, I.G. MacDonald, C. Rolland, H.G. Sol, F.J.M. van Asssche, A.A. Verrijn-Stuart, Information Systems Methodologies - A Framework for Understanding, 2nd Edition, Addison-Wesley, 1991.
Saeki, M., and K. Wen-yin, Specifying Software Specification and Design Methods. In: G. Wijers, S. Brinkkemper, T. Wasserman (Eds.), Proceedings of CAiSE’94, Lecture Notes in Computer Science 811, Springer Verlag, pp. 353-366, Berlin, 1994.
[Slooten 96]
Slooten, K. van and B. Hodes, Characterizing IS Development Projects. In: S. Brinkkemper, K. Lyytinen and R. Welke (Eds.), Method Engineering: Principles of Method Construction and Tool Support, Chapman & Hall, pp.29-44, 1996
[Song 95]
Song, X., A Framework for Understanding the Integration of Design Methodologies. In: ACM SIGSOFT Software Engineering Notes, Vol. 20, No. 1, pp. 46-54, 1995.
[Sorenseon 88]
Sorenson,P.G., J.P.Tremblay, A.J.McAllister, The Metaview System for Many Specifications Environements. In IEEE Software, Vol.30, No.3, pp.30-38, 1988.
[Ward 85]
Ward,P., S. Mellor, Structured Development for Real-time Systems, Yourdon Press, 1985.
Formalizing Materialization Using a Metaclass Approach ? Mohamed Dahchour??
Abstract. Materialization is a powerful and ubiquitous abstraction pattern for conceptual modeling. Intuitively, it relates a class of categories (e.g., models of cars) and a class of more concrete objects (e.g., individual cars). This paper formalizes the semantics of materialization using the metaclass approach of the TELOS data model. Formulas can be uniformly attached to classes, metaclasses, and meta-attributes to enforce integrity constraints and deductive rules relevant to materialization semantics. The paper also proposes some suggestions for extending TELOS to capture some materialization semantics which cannot be represented with the available constructs. Keywords: Object Orientation, Materialization Relationship, Metaclass, TELOS.
1
Introduction
Conceptual modeling is the activity of formalizing some aspects of the physical and social world around us for purposes of understanding and communication. Generic relationships are powerful abstraction constructs that help narrow the gap between concepts in the real world and their representation in conceptual models. For full benefit, these relationships should be made available in objectoriented languages and systems as primitives for developing conceptual models of applications. However, before their implementation, we believe that generic relationships should be first well formalized. This formalization will eliminate the possible ambiguities between similar relationships and will play an intermediate role between the informal description of a relationship and its factual implementation. This paper presents a formalization of materialization [PZMY94]. Materialization is a powerful and ubiquitous abstraction pattern. It is a semantic relationship between a class of abstract categories (e.g., models of cars) and a class of more concrete objects (e.g., individual cars). The semantics of materialization concern both classes and instances of these classes. Consequently, the formal specification of materialization must include both the specification of the ?
??
This work is part of the YEROOS (Yet another Evaluation and Research on ObjectOriented Strategies) project, principally based at the University of Louvain. See http://yeroos.qant.ucl.ac.be. University of Louvain, INGI (Department of Computer Science and Engineering), 2 Place Sainte-Barbe, 1348 Louvain-la-Neuve, Belgium, e-mail: [email protected]
B. Pernici, C. Thanos (Eds.): CAiSE’98, LNCS 1413, pp. 401–420, 1998. c Springer-Verlag Berlin Heidelberg 1998
402
Mohamed Dahchour
class and the instance levels in a coordinated manner [KS95]. Furthermore, constraints associated with generic relationships must be defined at the conceptual level, since they govern all instances of these relationships. We remove, therefore, the burden from the designers who otherwise would have to define these constraints for each realization of materialization. We use the metaclass approach of TELOS, a language for representing knowledge about information systems [MBJK90], to formalize materialization. TELOS has already been used to partially formalize semantics of partOf [MP93] and memberOf [MPK96] relationships. The metaclass approach has been used successfully to implement some generic relationships (see e.g., [HGPK94,KS95,GSR96]). Particularly, in our previous work [DPZ97], we have presented three metaclass approaches to implement generic relationships and in [DPZ96], we have used one of these approaches to implement materialization in an abstract target system. In this paper, we use the metaclass approach of TELOS for the formalization purpose. The paper is organized as follows. Section 2 gives an overview of materialization. Section 3 presents the main features of the TELOS data model, relevant to our formalization. Sections 4 and 5 formalize in detail the semantics of materialization at both the class and instance levels. Section 6 summarizes and concludes the paper.
2
Materialization
This section gives an overview of the materialization relationship and of its specific attribute propagation mechanisms. More detail can be found in [PZMY94]. 2.1
Intuitive definition
Intuitively, materialization relates a class of categories to a class of more concrete objects. Figure 1(a) shows a materialization relating two classes: class CarModel has two monovalued attributes (name and sticker price) and four multivalued attributes (#doors, eng size, auto sound, and special equip); class Car defines three
Formalizing Materialization Using a Metaclass Approach
403
monovalued attributes (manuf date, serial#, and owner). CarModel represents information typically displayed in the catalog of car dealers (namely, name and price of a car model, and lists of options for number of doors, engine size, sound equipment, and special equipment). Car represents information about individual cars (namely, manufacturing date, serial number, and owner identification). As in [PZMY94], we draw a materialization link as a straight line with a star ∗ on the side of the more concrete class. Figure 1(b) shows an instance FiatRetro of CarModel and an instance Nico’s Fiat of Car, of model FiatRetro. CarModel is the more abstract1 class and Car is the more concrete class of materialization CarModel—∗Car. Intuitively, this means that every concrete car (e.g., Nico’s Fiat) has exactly one model (e.g., FiatRetro), while there can be any number of cars of a given model. Further intuition about abstractness/concreteness is that each car is a concrete realization (or materialization) of a given car model, of which it “inherits” a number of properties in several ways. Nico’s Fiat thus directly inherits the name and sticker price of its model FiatRetro; this mechanism is called Type 1 attribute propagation. Nico’s Fiat has attributes #doors, eng size, and auto sound whose values are selections among the options offered by multivalued attributes with the same name in FiatRetro; this is called Type 2 attribute propagation. For example, the value {1200,1300} of eng size for FiatRetro indicates that each FiatRetro car comes with either eng size = 1200 or eng size = 1300 (e.g., 1200 for Nico’s Fiat). The value {airbag, alarm, cruise ctrl} of attribute special equip for FiatRetro means that each car of model FiatRetro comes with three pieces of special equipment: an airbag, an alarm system, and a cruise control system. Thus, Nico’s Fiat has three new attributes named airbag, alarm, and cruise ctrl, whose suppliers are, respectively, Acme, Burglar King, and Fiat. Other FiatRetro cars might have different suppliers for their special equipment. This mechanism is called Type 3 attribute propagation. In addition to those attributes propagated from the instance FiatRetro of class CarModel, Nico’s Fiat of course has a value for attributes manuf date, serial#, and owner of class Car. The semantics of attribute propagation is defined more precisely in Section 2.3. Abstract classes can materialize into several concrete classes. For example, data for a movie rental store could involve a class Movie, with attributes director, producer, and year, that materializes independently into classes VideoTape and VideoDisc (i.e., VideoTape∗—Movie—∗VideoDisc). VideoTapes and VideoDiscs could have attributes like inventory#, system (e.g., PAL or NTSC for VideoTape), language, availability (i.e., in-store or rented), and so on. Materializations can also be composed in hierarchies, where the concrete class of one materialization is also the abstract class of another materialization, and so on (e.g., Play—∗Setting—∗Performance). For the sake of space, this paper considers only simple materialization hierarchies A—∗C and abstract classes materializing in more than one concrete class as in C1∗—A—∗C2. A complete 1
The notion of abstractness/concreteness of materialization is distinct from the notion of abstract class of object models, where an abstract class is a class without instances, whose complete definition is typically deferred to subclasses.
404
Mohamed Dahchour
formalization of materialization, including composition of materializations, can be found in [Dah97]. 2.2
Semi-formal semantics
We now summarize the necessary elements for a semi-formal definition of materialization. Materialization is a binary relationship (A—∗C) between two classes A and C, where A is more abstract than C (or C is more concrete than A). Most real-world examples of materializations have cardinality [1, 1] on the side of the more concrete class C and cardinality [0, N ] on the side of the more abstract class A. Application semantics can further constrain the cardinality of the A-side to [Cmin , Cmax ], with the meaning that at least Cmin and at most Cmax concrete objects are associated with each abstract object. A a1 object facet
CarModel
* C
C1
...
FiatRetro object facet
an Cn
class facet
c11
... (a)
*
FiatRetro_Cars
Wild_2CV object facet
class facet
Two-faceted construct
cn1
Nico’s Fiat
: IsA : instance of
Car
Wild_2CV_Cars class facet
Two-faceted construct Guy’s 2CV
John’s Fiat
(b)
Fig. 2. Semantics of materialization.
The semantics of materialization is conveniently defined as a combination of usual is-a (generalization) and is-of (classification), and of a class/metaclass correspondence. Figure 2(a) shows how the semantics of materialization A—∗C is expressed with a collection of two-faceted constructs. Each two-faceted construct is a composite structure comprising an object, called the object facet, and an associated class, called the class facet. The object facet is an instance of the more abstract class A, while the class facet is a subclass of the more concrete class C. The semantics of materialization induce a partition of C into a family of subclasses {Ci }, such that each Ci is associated with exactly one instance of A. Subclasses Ci inherit attributes from C through the classical inheritance mechanism of the is-a link. They also “inherit” attributes from A, through the mechanisms of attribute propagation described in the next section. Objects of C, with attribute values “inherited” from an instance of A, are ordinary instances of the class facet associated with that instance of A. As in Figure 1, we draw classes as rectangular boxes and instances as rectangular boxes with rounded corners. Classification links (is-of) appear as dashed arrows, and generalization links (is-a) as solid arrows. To underline their double role, we draw a two-faceted construct as an object box adjacent to a class box. Figure 2(b) sketches the basic semantics of the materialization of Figure 1(a). The FiatRetro instance of CarModel is the object facet of a two-faceted construct, whose class facet is the subclass FiatRetro Cars of Car, describing all instances of
Formalizing Materialization Using a Metaclass Approach
405
Car with model FiatRetro. For users, Nico’s Fiat and John’s Fiat are instances of Car. Our semantics and its formalization describe them as ordinary instances of FiatRetro Cars. Wild 2CV is another instance of CarModel and Guy’s 2CV is an instance of class facet Wild 2CV Cars. 2.3
Attribute propagation
Attribute propagation from the more abstract class to the more concrete class of a materialization is precisely defined as a transfer of information from an abstract object to its associated class facet in a two-faceted construct, as illustrated in Figure 3. The three mechanisms of attribute propagation are defined precisely as follows: CarModel name (T1) sticker_price (T1) #doors (T2, mono) eng_size (T2, mono) auto-sound (T2, multi) special-equip (T3, Inst)
: IsA : instance of Nico’s Fiat name=Fiat-retro sticker_price=10.000 #doors=3 eng_size=1200 auto-sound={tape,radio} airbag = Acme alarm = Burglar_King cruise = Fiat manuf_date=1/1/95 serial#=123 owner=NICO
Fig. 3. Attribute propagation between CarModel and Car.
1. For users, Type 1 propagation characterizes the plain transfer of an attribute value from an instance of the abstract class to instances of the concrete class. In our semantics, the value of a (monovalued or multivalued) attribute is propagated from an object facet to its associated class facet as a class attribute (i.e., an attribute whose value is the same for all instances of the class facet). For example, monovalued attributes name and sticker price of CarModel are Type 1 in materialization CarModel—∗Car (see Figure 3). Their value in object facet FiatRetro (Fiat-retro and 10.000, respectively) propagates as value of class attributes with the same name in class facet FiatRetro Cars. 2. For users, Type 2 propagation concerns multivalued attributes of the more abstract class A. Their value for an instance of A determines the type, or domain, of instance attributes with the same name, monovalued or multivalued, in the associated class facet. Again, our semantics go through abstract objects and associated class facets.
406
Mohamed Dahchour
An example of the monovalued case is exhibited by attribute eng size of CarModel. Its value {1200,1300} for the FiatRetro object facet is the domain of values for a monovalued instance attribute with the same name eng size of the associated class facet FiatRetro Cars. Thus, each FiatRetro car comes either with eng size = 1200 or with eng size = 1300. An example of the multivalued case is exhibited by attribute auto sound of CarModel. Its value {tape, radio} indicates that each FiatRetro car comes with either tape, or radio, or both, or nothing at all as auto sound. The associated class facet FiatRetro Cars has a multivalued instance attribute auto sound with the powerset P({tape, radio}) as its type. 3. Type 3 propagation is more elaborate. It also concerns multivalued attributes of the more abstract class A, whose value is always a set of strings. Each element in the value of an attribute for object facet a generates a new instance attribute in the class facet associated with a. The type of generated attributes must be specified in the definition of the materialization. For example, attribute special equip of CarModel propagates with Type 3 to Car. Its value {airbag, alarm, cruise ctrl} for object FiatRetro generates three new monovalued instance attributes of type string, named airbag, alarm, and cruise ctrl, for the associated class facet FiatRetro Cars.
3
The TELOS data model
This section gives a general view of the main features of the TELOS data model relevant to our formalization. More details about TELOS can be found in [MBJK90]. TELOS is actually supported by the ConceptBase system [JJS96]. TELOS is a language for representing knowledge about information systems. TELOS knowledge bases are collections of propositions. Each proposition p is a three-tuple where from, label, and to denote the source, label, and destination of the proposition, respectively. These elements can be accessed through the functions From(p), Label(p), and To(p). TELOS propositions are either individuals or attributes. Individuals represent what are called objects (e.g., the individual book OOSC2ed) and classes (e.g., Book) in usual object models. While attributes represent binary relationships between individuals or other relationships. An example of an attribute is [OOSC2ed, author, “B. Meyer”]. Propositions can be classified in an arbitrary number of classification levels where each proposition is an instance of one or more generic propositions called classes. Classes that are themselves propositions must be in their turn instances of more generic classes, and so on. For example, OOSC2ed and [OOSC2ed, author, ‘B. Meyer’] are instances of Book and [Book, author, Person], respectively. The so-called ω-classes can have instances along more than one level of classification. For example, Proposition has all propositions as instances and Class has all generic propositions as instances. The following example shows the use of the different features above. The TELL operation is used to add a new proposition in the knowledge base (i.e., create new objects in the terminology of usual object models) or to add new attributes to an already defined one.
Formalizing Materialization Using a Metaclass Approach TELL TOKEN MT-93-#1 In BorrowedDocument WITH author firstAuthor: “C. Marcos”; secondAuthor: “M. Cilia”; title : “A SDM approach for the Prototyping of IS” borrowed : Yes; borrower : “John” outDate : “05/06/97:9H” inDate : “05/06/97:18H” END
407
TELL CLASS Document IN Class WITH attribute author: Person; title: String; ... END TELL CLASS BorrowedDocument IsA Document, IN Class WITH attribute borrowed: String; borrower: Person; outDate: Date; inDate: Date; END
Fig. 4. TELOS definition of instances, classes, and attributes.
Figure 4 shows, on the left side, the individual document MT-93-#1 that is declared as an instance (via the IN clause) of the class BorrowedDocument defined on the right side of the figure as an instance of the metaclass Class and as a specialization of Document. The WITH clause introduces the list of attributes. For example, the two first attributes of MT-93-#1, firstAuthor and secondAuthor, are two instances of the attribute class author. The attribute [MT93-#1, firstAuthor, “C. Marcos”] is an instance of [Document, author, Person] in exactly the same sense that MT-93-#1 is an instance of Document. The third attribute of MT-93-#1 has no external label and it is an instance of the title class. Labels of such attributes are automatically generated by the system. In Telos, a proposition may be an instance of more than one class (multiple classification). For instance, MT-93-#1 can be an instance of both classes MasterThesisDocument and RestrictedDocument which stands for a collection of documents that are not allowed to go out the library. Meta-attributes. The first-class status of attributes and the ability to define attributes and meta-attributes are very important in TELOS. Figure 5 shows an example of meta-attributes which are needed to define common properties of the various resource classes. These meta-attributes are introduced through the metaclass ResourceClass. In this example, source, what, available, who, from, and until are meta-attributes which may be instantiated for ResourceClass instances. The class BorrowedDocument is declared now as an instance of ResourceClass on the right side of Figure 5 and its attribute borrower is an instance of the meta-attribute who.
TELL CLASS ResourceClass WITH attribute source: Class; what: Class; available: Class; who: Class; from: Class; until: Class; END
TELL CLASS BorrowedDocument IN ResourceClass WITH source author: Person; who what borrower: Person; title: String; from available outDate: Date; borrowed: String; until inDate: Date; END
Fig. 5. Definition of meta-attributes.
408
Mohamed Dahchour
As another example of use of meta-attributes, Figure 6 gives the definition of the meta-attribute single that restricts its instances to (at most) a single value [MBJK90]. The right side of Figure 6 shows an example of use of the meta-attribute single: we restrict the borrower of a BorrowedDocument to a single value by declaring it as an instance of single. The meta-attribute single is defined in the metaclass Class and it is inherited by BorrowedDocument by declaring BorrowedDocument as instance of Class. Note that by default a TELOS attribute such as author: Person of Figure 5 can have several instances. If we want to restrict the attribute value, we have to use something like the meta-attribute single. Therefore, the declaration of attributes in TELOS should not be confused with that of the usual object data models. TELL CLASS Class WITH attribute single: Class integrityConstraint single Cnstr: $ (∀ u/Class!single)(∀ p,q/Proposition) (p in u) ∧ (q in u) ∧ From(p) = From(q) ⇒ (p = q) $ END
TELL CLASS BorrowedDocument IN ResourceClass, Class WITH ... who, single borrower: Person; ... END
Fig. 6. Definition of the single meta-attribute and its use.
Constraints, rules, and metaformulas. TELOS supports an assertion sublanguage to specify integrity constraints and deductive rules. Constraints and rules are formulas that appear as attribute values of propositions. They specify the behavioral part of the objects to which they are attached. Constraints are assertions that control the information supplied by users, while deductive rules are assertions that enforce new facts. For example, the integrity constraint c1 of the definition of Figure 7 ensures that the out of date for a borrowed document x must always be less than its return date. The constraint c2 ensures that a given document x cannot be borrowed by two persons at overlapping dates2 . The deductive rule states that once a person p borrows a certain document x, the system automatically derives the fact (x.borrowed = Yes), indicating that the document is actually borrowed. The “x/C” notation is read “x is an instance of C”. TELL CLASS BorrowedDocument IN Class WITH integrityConstraint c1: $ (∀ x/BorrowedDocument) (x.outDate ≤ x.inDate)$; c2: $ (∀ x/BorrowedDocument)(∀ p1, p2/Person) (x.borrower=p1) [at t1] ∧ (x.borrower=p2) [at t2] ∧ (t1 overlaps t2) ⇒ (p1 = p2) $ deductiveRule : $ (∀ x/BorrowedDocument)(∀ p/Person) (x.borrower = p) ⇒ (x.borrowed = Yes) $ END
Fig. 7. Definition of constraints and deductive rules. 2
TELOS also supports an explicit representation of time which is not presented in this paper (see [MBJK90]).
Formalizing Materialization Using a Metaclass Approach
409
In traditional modeling languages, a formula is defined for a given class to constrain the behavior of only the instances of this class. In TELOS, the socalled metaformulas can be associated to a given metaclass to specify the behavior of both the instances of this metaclass and the instances of its instances. As an example, the constraint attached to the metaclass Class on the left side of Figure 6 is a metaformula that manipulates p and q that are instances of instances of Class!single. To manipulate attributes and their values in definitions of formulas, we need the following functions where the time constraints are omitted [MBJK90]: 1. The dot function x.l evaluates to the set of values of the attributes of proposition x which belong to the attribute class labeled l. 2. The hat function x b l evaluates to the set of values of the attributes of proposition x with label l. 3. The bar function x|l evaluates to the set of attribute propositions with source x which are instances of the attribute class labeled l. 4. The exclamation function x!l evaluates to the set of attribute propositions with source x and label l.
4
Formalizing the class level semantics of materialization
In this section we formalize the class level semantics of the materialization relationship by means of two metaclasses AbstractClass and ConcreteClass that represent, respectively, abstract and concrete classes in materialization hierarchies. TELL CLASS AbstractClass In Class WITH attribute materializes: ConcreteClass END
TELL CLASS ConcreteClass In Class WITH attribute, single materOf: AbstractClass END
TELL CLASS AbstractClass WITH deductiveRule materDedRule: $ (∀ A/AbstractClass)(∀ C/ConcreteClass) (C ∈ A.materializes) ⇒ (C.materOf = A) $ END
Fig. 8. Definition of AbstractClass and ConcreteClass metaclasses.
Figure 8 shows the definitions of the AbstractClass and ConcreteClass metaclasses. We declare AbstractClass as instance of the predefined metaclass Class. AbstractClass contains one meta-attribute whose label is materializes and destination is ConcreteClass. In the middle of Figure 8, we declare the metaclass ConcreteClass that plays the inverse role of AbstractClass. ConcreteClass contains one meta-attribute whose label is materOf and destination is AbstractClass. The materOf meta-attribute is constrained to be of a single value, meaning that a given concrete class has only one associated abstract class. On the right side of Figure 8, we add the deductive rule materDedRule to the AbstractClass metaclass to specify that once a given class A is declared as an abstract class which materializes in a given concrete class C, the system automatically induces the fact (C.materOf = A) which means that C is a materialization
410
Mohamed Dahchour
of the abstract class A. A similar deductive rule can be associated with the ConcreteClass metaclass to play the dual role. 4.1
Definition of the materialization characteristics
Materialization characteristics are formalized as attributes of the meta-attribute materializes. To be able to attach properties to materializes, we have to declare this later as a metaclass as shown in Figure 9. TELL CLASS AbstractClass!materializes In Class, Attribute WITH attribute cardinality: CardType, inhAttrT1: Attribute-1Def, inhAttrT2: Attribute-2Def, inhAttrT3: Attribute-3Def TELL CLASS CardType In Class WITH attribute min: Integer; max: Integer integrityConstraint minmaxCard: $(∀ c/CardType) (c.min ≤ c.max)$ END
TELL CLASS Attribute-1Def In String END TELL CLASS Attribute-2Def In Class WITH attribute attrName: String; derivAttr: String {* mono or multi *} END TELL CLASS Attribute-3Def In Class WITH attribute attrName: String; genAttrType: TypeDef; END TELL CLASS TypeDef In Class WITH END
Fig. 9. Definition of the materialization characteristics.
In Figure 9, we apply the “!” symbol to AbstractClass to access the attribute materializes itself. The figure shows the following characteristics: cardinality denotes the cardinality of an abstract class regarding a concrete class. The trivial associated constraint minmaxCard states that the minimal cardinality is always less than the maximal one. The remaining attributes labeled inhAttrT1, inhAttrT2, and inhAttrT3 specify propagation modes for attributes of the abstract class to the corresponding concrete class. Definitions of their destinations (i.e., domains) are given on the right side of the figure: 1. Attribute-1Def is the name of an attribute propagating with Type 1; 2. Attribute-2Def gives, for an attribute propagating with Type 2, its name and the kind derivedAttr (monovalued or multivalued) of the derived instance attribute 3 ; 3. Attribute-3Def gives the name of an attribute propagating with Type 3 and the value type genAttrType (TypeDef). Note that the meta-attribute inhAttrT1 (resp., inhAttrT2 and inhAttrT3) can be instantiated in application with as many Type 1 (resp., Type 2 and Type 3) attributes as needed. 4.2
Example of use of generic semantics
As an example, Figure 10 shows how the CarModel—∗Car materialization is formalized by invoking the generic semantics. 3
The {* . . . *} notation denotes a TELOS comment.
Formalizing Materialization Using a Metaclass Approach TELL CLASS CarModel In Class WITH attribute name: String; sticker price: Integer; #doors: Integer; eng size: Integer; auto sound: String; special equip: String END
411
TELL CLASS Car In Class WITH attribute manuf date: Date; serial# : Integer; owner: String END
(a) TELL CLASS CarModel In AbstractClass WITH materializes materializesCar: Car END TELL CLASS Car In ConcreteClass WITH END TELL CLASS CarModel!materializesCar In Class WITH cardinality: 0 N inhAttribT1 T11: name; T12: sticker price inhAttribT2 T21: T2Doors; T22: T2EngSize; T23: T2AutoSound inhAttribT3. T31: T3SpecialEquip END TELL TOKEN 0 N In CardType WITH min: 0; max: 100 {* 100 denotes here the ∞ (N) symbol *} END
TELL TOKEN name In Attribute-1Def WITH END TELL TOKEN sticker price In Attribute-1Def WITH END TELL TOKEN T2Doors In Attribute-2Def WITH attrName: “#doors”; derivAttr: “mono” END TELL TOKEN T2EngSize In Attribute-2Def WITH attrName: “eng size”; derivAttr: “mono” END TELL TOKEN T2AutoSound In Attribute-2Def WITH attrName: “auto sound”; derivAttr: “multi” END TELL TOKEN T3SpecialEquip In Attribute-3Def WITH attrName: “special equip”; genAttrType: String; END TELL CLASS String IN TypeDef END
(b) Fig. 10. Materialization CarModel—∗Car.
In Figure 10 (a), the classes CarModel and Car are created as ordinary classes, independently of the notion of materialization. To take into account the materialization relationship between CarModel and Car, we declare in Figure 10 (b) CarModel as an instance of AbstractClass and Car as an instance of ConcreteClass. During the creation of CarModel we instantiate the meta-attribute materializes of AbstractClass by materializesCar. Note that there is no need to instantiate the meta-attribute materOf of ConcreteClass during the creation of Car. This will be automatically achieved by the deductive rule materDedRule of Figure 8 (b). In the attribute CarModel!materializesCar we specify that: the cardinality of CarModel regarding Car is [0:N]; name and sticker price propagate with Type 1; #doors and eng size both propagate with Type 2 and each produces a monovalued instance attribute, while auto sound produces, also with Type 2, a multivalued instance attribute; special equip generates, with Type 3, new instance attributes of type String. Generic semantics of materialization given above also hold for abstract classes that materialize in more than one concrete class such as a hierarchy of materializations C1∗—A—∗C2. In such a case, we create A as an instance of AbstractClass and instantiate the attribute materializes into two attributes materializesC1 and materializesC2 with destinations C1 and C2, respectively. Then, we declare A!materializesC1 and A!materializesC2 as instances of the metaclass Class to capture different characteristics of materializations A—∗C1 and A—∗C2, respectively.
412
4.3
Mohamed Dahchour
Constraints related to inheritance attributes
This section defines two constraints related to inheritance attributes. The first one expresses the membership of inheritance attributes to abstract classes and the second states that Type 2 attributes must be multivalued. Inheritance attributes are members of abstract classes. All inheritance attributes appearing in the definition of the meta-attribute AbstractClass!materializes must be attributes of the more abstract class. For instance, the Type 1 attributes (name, sticker price), the Type 2 attributes (#doors, eng size, and auto sound), and the Type 3 attribute (special equip) of the Figure 10 (b) must be attributes of the abstract class CarModel. Figure 11 shows, respectively, the constraints T1Attr Cnstr, T2Attr Cnstr, and T3Attr Cnstr that express the membership of the inheritance attributes to the abstract class of a materialization relationship.
TELL CLASS AbstractClass!materializes TELL CLASS AbstractClass!materializes TELL CLASS AbstractClass!materializes WITH integrityConstraint WITH integrityConstraint WITH integrityConstraint T2Attr Cnstr: T3Attr Cnstr: T1Attr Cnstr: $ (∀ M/AbstractClass!materializes) $ (∀ M/AbstractClass!materializes) $ (∀ M/AbstractClass!materializes) (∀ A/AbstractClass) From(M)=A ⇒ (∀ A/AbstractClass) From(M)=A ⇒ (∀ A/AbstractClass) From(M)=A ⇒ (∀ T1/Attribute-1Def)(∀ S1/String) (∀ T2/Attribute-2Def)(∀ S2/String) (∀ T3/Attribute-3Def)(∀ S3/String) (T1 ∈ M.inhAttrT1 ∧ To(T1)=S1 (T2 ∈ M.inhAttrT2) ∧ (S2 = T2.attrName) (T3 ∈ M.inhAttrT3) ∧ (S3 = T3.attrName) ⇒ (∃ p/Proposition) ⇒ (∃ p/Proposition) ⇒ (∃ p/Proposition) (p ∈ A | attribute) (p ∈ A | attribute) ∧ Label(p)=S1 $ (p ∈ A | attribute) ∧ Label(p)=S2 $ ∧ Label(p)=S3 $ END END END
Fig. 11. Membership metaconstraints of inheritance attributes.
Type 2 attributes are multivalued. Further the constraints related to membership of the inheritance attributes to the abstract classes, Type 2 attributes (e.g., #doors, eng size, and auto sound of Figure 10 (b)) must be multivalued: they must have more than one value at the instance level (e.g., in FiatRetro of Figure 3, #doors has two values 1200 and 1300). TELL CLASS Class WITH TELL CLASS AbstractClass WITH attribute integrityConstraint multivalued: Class T2attr are multivalued: $ (∀ A/AbstractClass)(∀ M/AbstractClass!materializes) integrityConstraint From(M)=A ⇒ multivalued Cnstr: (∀ T2/Attribute-2Def)(∀ S2/String) $ (∀ attm/Class!multivalued)(∃ p,q/Proposition) (T2 ∈ M.inhAttrT2) ∧ (S2 = T2.attrName) (p in attm) ∧ (q in attm) ∧ From(p) = From(q) ⇒ (∃ p/Proposition) (p ∈ A | attribute) ∧ Label(p)=S2 ⇒ (p 6= q) ⇒ (p in Class!multivalued) $ $ END END
Fig. 12. Metaconstraint related to values of Type 2 inheritance attributes.
The left side of Figure 12 shows the definition of the constraint ensuring the above requirements. The Type 2 attributes are declared as instances of the metaattribute Class!multivalued we define on the right side of Figure 12. This figure
Formalizing Materialization Using a Metaclass Approach
413
shows that the source and the destination of the attribute labeled multivalued are of type Class. The constraint associated with multivalued states that for every instance attm of Class!multivalued, there are at least two distinct instances p and q with common source4 . According to the constraint of Figure 12, Figure 10(a) must be revised: the Type 2 attributes (#doors, eng size, and auto sound) must have been declared as instances of the meta-attribute multivalued. 4.4
Cardinality constraints
In this section we formalize the cardinality constraints of materialization. Definition of the [0:N] and [1:1] cardinalities. Figure 13 shows definitions of two constraints card0 NCnstr and card1 1Cnstr which formalize, respectively, the cardinalities [0:N] associated with abstract classes and [1:1] associated with concrete classes. In Figure 13, we manipulate the abstract and concrete objects (e.g., a and c) that are, respectively, instances of AbstractObject and ConcreteObject classes we define in the next section. The last implication of the constraint card0 NCnstr is always evaluated to TRUE, meaning that we tolerate both the existence and the absence of a concrete instance c for a given abstract object a. The card1 1Cnstr definition is composed of two parts: the existence part which states that for each concrete object there is an associated abstract object and the uniqueness part stating that there is one and only one associated abstract object. TELL CLASS AbstractClass!materializes WITH integtityConstraint card0 NCnstr: $ (∀ M/AbstractClass!materializes) (∀ A/AbstractClass)(∀ C/ConcreteClass) From(M)=A ∧ To(M)=C ∧ (M.cardinality = 0 N) ⇒ (∀ a/AbstractObject) (a in A) ⇒ (∃ c/ConcreteObject) (c in C) ∧ (c ∈ a.materializes) ⇒ (TRUE) $ END TELL TOKEN 0 N In CardType WITH min: 0; max: 100 {* 100 denotes here the ∞ (N) symbole *} END
TELL CLASS ConcreteClass!materOf WITH integtityConstraint card1 1Cnstr: $ (∀ M/ConcreteClass!materOf)(∀ A/AbstractClass)(∀ C/ConcreteClass) From(M)=C ∧ To(M)=A ∧ (M.cardinality = 1 1) ⇒ (∀ c/ConcreteObject) (c in C) ⇒ (∃ a/AbstractObject) (a in A) ∧ (a ∈ c.materOf) {* existence *} ∧ (∀ a1, a2/AbstractObject) (a1 in A) ∧ (a1 ∈ c.materOf) ∧ (a2 in A) ∧ (a2 ∈ c.materOf) ⇒ (a1 = a2)) {* uniqueness *} $ END TELL TOKEN 1 1 In CardType WITH min: 1; max: 1 END
Fig. 13. Definition of the [0:N] and [1:1] cardinality constraints.
Definition of the [Cmin , Cmax ] cardinality. As for the [Cmin , Cmax ] cardinality (e.g., [3,6]), which may be associated at the side of abstract classes, there is no way, in Telos, for its formalization. To deal with this category of cardinality, we define a new predicate Length(X, Class, Assertion, Integer) with the following meaning: the last argument is the number of elements X, instances of the second 4
The meta-attribute multivalued is defined in the same spirit as the meta-attribute single (see Figure 6).
414
Mohamed Dahchour
argument, satisfying the assertion given in the third argument. Formally, we can write: Length(x, C, P, N) ≡ # {x/C Holds(P(x))}= N The predicate Holds is true whenever its argument is an assertion that follows from the knowledge base [MBJK90]. The “#” symbol is applied to a given set and denotes the number of its elements. As a result of this new predicate, the constraint related to the [Cmin , Cmax ] cardinality will be as shown in Figure 14.
TELL CLASS AbstractClass!materializes WITH integtityConstraint cardMin MaxCnstr: $ (∀ M/AbstractClass!materializes) (∀ A/AbstractClass)(∀ C/ConcreteClass) From(M)=A ∧ To(M)=C ∧ 1≤M.cardinality.min≤M.cardinality.max ∧ M.cardinality.max≥2 ⇒ (∀ a/AbstractObject) (a in A) ∧ (a in AbstractObject) ⇒ Length(c, ConcreteObject, (c in C) ∧ (c ∈ a.materializes), K) ∧ (Min Max.min ≤ K ≤Min Max.max) $ END
Fig. 14. Definition of the constraint related to the cardinality [Cmin , Cmax ].
5
Formalizing the instance level semantics of materialization
This section describes the instance level semantics of materialization by means of two classes AbstractObject and ConcreteObject that represent, respectively, abstract objects and concrete objects, instances of instances of the metaclasses AbstractClass and ConcreteClass, respectively.
TELL CLASS AbstractObject In Class WITH attribute classFacet: Class materializes: ConcreteObject END
TELL CLASS ConcreteObject In Class WITH attribute, single materOf: AbstractObject END
TELL CLASS AbstractObject WITH deductiveRule materDedRule: $ (∀ a/AbstractObject)(∀ c/ConcreteObject) (c ∈ a.materializes) ⇒ (c.materOf = a) $ END
Fig. 15. Definitions of AbstractObject and ConcreteObject classes.
The definitions of the classes AbstractObject and ConcreteObject are depicted in Figure 15. On its left side, the figure shows that each abstract object has an associated class facet, denoted classFacet. An abstract object has also an attribute materializes whose domain is ConcreteObject and each concrete object has an attribute materOf whose domain is AbstractObject. For reason of uniformity, the same names materializes and materOf are used both for the description of the class level and the instance level of materialization semantics.
Formalizing Materialization Using a Metaclass Approach
415
The attribute materOf is declared as an instance of single to assert that a concrete object is materialization of only one abstract object. Finally, as in Figure 8, we give on the right side of Figure 15 a deductive rule that automatically derives from the fact (c ∈ a.materializes) another fact (c.materOf = a) indicating that c is a materialization of a. 5.1
Constraints related to the abstract objects
Figure 16 shows two constraints concerning the abstract objects. On the left side we define the abstractObj Cnstr constraint which expresses that instances of instances of AbstractClass must be instances of AbstractObject. On the right side, the ObjClassFacet Cnstr constraint means that each abstract object a has one and only one (denoted by the “∃!” symbole) associated class facet regarding a given materialization. The constraint also shows that the class facet must be a subclass of the concrete class C that is a materialization of the abstract class A, the class of a.
TELL CLASS AbstractClass WITH integrityConstraint abstractObj Cnstr: $ (∀ A/AbstractClass)(∀ a/Class) (a in A) ⇒ (a in AbstractObject) $ END
TELL CLASS AbstractClass WITH integrityConstraint ObjClassFacet Cnstr: $ (∀ A/AbstractClass)(∀ C/ConcreteClass) (∀ a/AbstractObject) (a in A) ∧ (C ∈ A.materializes) ⇒ (∃! Cf/Class) (Cf isA C) ∧ (Cf ∈ a.classFacet) $ END
Fig. 16. Constraints associated with the abstract objects.
The ObjClassFacet Cnstr constraint implicitly means that, each time the user creates an abstract instance a, he/she also must create explicitly its associated class facet to ensure the constraint. We think that it would be more reasonable and more natural to implicitly generate the class facet of a given abstract object. For this, we propose to extend the definition of the deductive rule. The actual definition of a deductive rule allows only the possibility to derive facts that are logical expressions. The extension we propose consists of the specification, on the right side of a deductive rule, of some operations we wish to be executed at run time as for the rule-based system MARVEL [KFP88]. Such an extended rule will be of the pattern: < preconditions > ⇒ < usualf acts > [and < operations >] where < usualf acts > is the ordinary part we find on the right side of a usual deductive rule and < operations > stands for operations we wish to be automatically executed by the system when the part < preconditions > is satisfied. The ObjClassFacet Cnstr constraint on the right side of Figure 16 becomes the following extended deductive rule: This deductive rule means that for each abstract object a, instance of an abstract class A which materializes in C, the system automatically generates a
416
Mohamed Dahchour TELL CLASS AbstractClass WITH deductiveRule ObjClassFacetExt Cnstr: $ (∀ A/AbstractClass)(∀ C/ConcreteClass) (∀ a/AbstractObject) (a in A) ∧ (C ∈ A.materializes) ⇒ TELL CLASS Cf in Class, isA C WITH END ∧ (Cf ∈ a.classFacet) $ END
class Cf as an instance of Class and as a subclass of C. The system induces then the fact (Cf ∈ a.classFacet) meaning that Cf is a potential class facet of a. 5.2
Constraints related to the concrete objects
Figure 17 shows two constraints regarding the concrete objects. On the left side, the concObj Cnstr constraint expresses that instances of instances of ConcreteClass must be instances of ConcreteObject. On the right side, the concObjClassFacet Cnstr constraint expresses that all concrete objects c, instances of a given concrete class C, and materializing a given abstract object a, must be instances of the class facet associated with a. For example, Nico’s Fiat that is instance of Car and that materializes FiatRetro must be instance of FiatRetro Cars, the class facet associated with FiatRetro (see Figure 2).
TELL CLASS ConcreteClass WITH integrityConstraint concObj Cnstr: $ (∀ C/ConcreteClass)(∀ c/Class) (c in C) ⇒ (c in ConcreteObject) $ END
TELL CLASS ConcreteClass WITH integrityConstraint concObjClassFacet Cnstr: $ (∀ A/AbstractClass)(∀ C/ConcreteClass) (C ∈ A.materializes) ⇒ (∀ c/ConcreteObject) (∀ a/AbstractObject) (a in A) ∧ (Cf ∈ a.classFacet) ∧ (Cf isA C) ∧ (c in C) ∧ (c ∈ a.materializes) ⇒ (c in Cf) $ END
Fig. 17. Constraints associated with the concrete objects.
5.3
Constraints related to attribute propagation
Propagation of Type 1 attributes. Figure 18 shows, on its left side, the constraint T1AttrInClassFacet Cnstr which imposes the Type 1 attributes of an abstract class A to be attributes of each class facet associated with each instance a of A. The right side of Figure 18 shows the constraint T1AttrAreClassAttr Cnstr which states that Type 1 attributes are class attributes5 in class facets: for a given Type 1 attribute T1, if the value of T1 in a given abstract object a is v then all the instances of the class facet associated with a will come with the same value v. Another alternative to deal with Type 1 attributes is the use of delegation mechanism [Lie86] that would permit concrete instances to access directly the 5
we mean here the usual sense of “class attributes” as opposed to “instance attributes”.
Formalizing Materialization Using a Metaclass Approach
417
Type 1 attribute values in the abstract instance a, without storing it redundantly in class facets. Unfortunately, TELOS does not provide facilities for using the delegation mechanism.
TELL CLASS AbstractClass WITH integrityConstraint T1AttrInClassFacet Cnstr: $ (∀ M/AbstractClass!materializes) (∀ A/AbstractClass) (∀ C/ConcreteClass)(∀ T1/Attribute-1Def) (∀ S1/String) (∀ D/Domain) (C ∈ A.M) ∧ (T1 ∈ M.inhAttrT1) ∧ To(T1)=S1 ⇒ (∃ p/Proposition) (p ∈ A | attribute) ∧ Label(p)=S1 ∧ To(p)=D ⇒ (∀ a/AbstractObject) (∀ Cf/Class) (a in A) ∧ (Cf ∈ a.classFacet) ∧ (Cf isA C) ⇒ (∃ q/Proposition) (q ∈ Cf | attribute) ∧ Label(q)=S1 ∧ To(q)=D $ END
TELL CLASS AbstractClass WITH integrityConstraint T1AttrAreClassAttr Cnstr: $ (∀ M/AbstractClass!materializes) (∀ A/AbstractClass) (∀ C/ConcreteClass)(∀ T1/Attribute-1Def) (∀ S1/String) (∀ D/Domain) (C ∈ A.M) ∧ (T1 ∈ M.inhAttrT1) ∧ To(T1)=S1 ⇒ (∃ p/Proposition) (p ∈ A | attribute) ∧ Label(p)=S1 ∧ To(p)=D ⇒ (∀ a/AbstractObject) (∀ Cf/Class) (a in A) ∧ (Cf ∈ a.classFacet) ∧ (Cf isA C) ⇒ (∀ q/Proposition) (q ∈ Cf | attribute) ∧ Label(q)=S1 ∧ To(q)=D ⇒ (∀ u,v,c/Proposition) (c in Cf) ∧ (u in q) ∧ From(u)=c ∧ To(u)=v ⇒ (v = a.S1) $ END
Fig. 18. Constraints associated with Type 1 attribute propagation.
Propagation of Type 2 attributes. The constraint T2AttrInClassFacet Cnstr of the left side of Figure 19 imposes the Type 2 attributes of an abstract class A to be also attributes of each class facet associated with each instance a of A. TELL CLASS AbstractClass WITH TELL CLASS AbstractClass WITH integrityConstraint integrityConstraint T2ValuePropag Cnstr: T2AttrInClassFacet Cnstr: $ (∀ M/AbstractClass!materializes) (∀ A/AbstractClass) $ (∀ M/AbstractClass!materializes)(∀ A/AbstractClass) (∀ C/ConcreteClass)(∀ T2/Attribute-2Def) (∀ C/ConcreteClass) (∀ T2/Attribute-2Def)(∀ S2/String) (∀ S2/String)(∀ D/Domain) (C ∈ A.M) ∧ (T2 ∈ M.inhAttrT2) ∧ (S2 = T2.attrName) ⇒ (C ∈ A.M) ∧ (T2 ∈ M.inhAttrT2) ∧ (S2 = T2.attrName) (∀ a/AbstractObject)(∀ Cf/Class)(∀ q/Proposition) ⇒ (a in A) ∧ (Cf ∈ a.classFacet) ∧ (Cf isA C) ∧ (∃ p/Proposition) (q ∈ Cf | attribute) ∧ Label(q)=S2 ⇒ (p ∈ A | attribute) ∧ Label(p)=S2 ∧ To(p)=D [ (T2. derivAttr = “mono”) ⇒ (q in Class!single) ∧ ⇒ (∀ a/AbstractObject)(∀ Cf/Class) ((∀ u,v,c/Proposition) (c in Cf) ∧ (u in q) ∧ From(u)=c ∧ To(u)=v ⇒ (a in A) ∧ (Cf ∈ a.classFacet) ∧ (Cf isA C) (v ∈ a.S2) ) ] ∧ ⇒ (∃ q/Proposition) (q ∈ Cf | attribute) ∧ [ (T2. derivAttr = “multi”) ⇒ Label(q)=S2 ∧ To(q)=D $ (∀ u,v,c/Proposition) (c in Cf) ∧ (u in q) ∧ From(u)=c ∧ To(u)=v ⇒ END (v ∈ a.S2) ] $ END
Fig. 19. Constraints associated with Type 2 attribute propagation.
The domain of the Type 2 attributes in class facets is the same as in the abstract class, but it will be, next, restricted to only a set of fixed values. The constraint responsible of this restriction is T2ValuePropag Cnstr as given on the right side of Figure 19. This constraint expresses that for each concrete instance c, instance of a class facet Cf, the value of its Type 2 attribute S2 must belong to the set of values of S2 in the abstract object a that materializes in c. If S2 is monovalued, then S2 must have only one value in c, otherwise S2 can have several values. For instance, the Type 2 attribute “#doors:Integer” of CarModel must be also an attribute of the class facet FiatRetro Cars associated with FiatRetro, instance
418
Mohamed Dahchour
of CarModel. The domain of #doors in FiatRetro Cars is also Integer, but its value in Nico’s Fiat, instance of FiatRetro Cars, must be restricted to either 3 or 5: the two possible values fixed by the abstract instance FiatRetro. Propagation of Type 3 attributes. Figure 20 shows the T3ValuePropag Cnstr constraint that formalizes the propagation of Type 3 attributes. It shows that each value V3 of a given Type 3 attribute in an abstract object a is a new attribute of the class facet Cf associated with a. The domain of V3 in Cf is D which a user can supply in advance by using the attribute genAttrType associated with the definition of Type 3 attribute (Attribute-3Def). TELL CLASS AbstractClass WITH integrityConstraint T3ValuePropag Cnstr: $ (∀ M/AbstractClass!materializes)(∀ A/AbstractClass) (∀ C/ConcreteClass)(∀ T3/Attribute-3Def) (∀ S3/String)(∀ D/TypeDef) (C ∈ A.M) ∧ (T3 ∈ M.inhAttrT3) ∧ (S3 = T3.attrName) ∧ (T3.genAttrType = D) ⇒ (∃ p/Proposition) (p ∈ A | attribute) ∧ Label(p)=S3 ⇒ (∀ a/AbstractObject)(∀ Cf/Class) (∀ V3/Class) (a in A) ∧ (Cf ∈ a.classFacet) ∧ (V3 ∈ a.S3) ⇒ (∃ q/Proposition) (q ∈ Cf | attribute) ∧ Label(q)=V3 ∧ To(q)=D $ END
Fig. 20. Constraints associated with Type 3 attribute propagation.
6
Conclusion
This paper has presented a formalization of materialization, a new and ubiquitous abstraction pattern relating a class of abstract categories and a class of more concrete objects. Materialization allows the definition of new and powerful attribute propagation (or inheritance) mechanisms from the abstract class to the concrete class. Our formalization was carried out using the metaclass approach of TELOS. Two metaclasses AbstractClass and ConcreteClass were built as templates to capture the semantics of materialization at the class level and two additional classes AbstractObject and ConcreteObject were defined to capture the semantics of materialization at the instance level. The metaclasses AbstractClass and ConcreteClass define the meta-attributes materializes and materOf, respectively. Thanks to the class status of TELOS attributes, the meta-attributes materializes and materOf are declared as ordinary metaclasses to which we attached all characteristics of materialization relationship. Various constraints have been uniformly defined at the levels of classes, metaclasses, and attributes to ensure the semantics of materialization at both the class and the instance levels in a coordinated manner. The metaclass approach allowed us to define the semantics of materialization as a unit which is independent of any implementation consideration. Conse-
Formalizing Materialization Using a Metaclass Approach
419
quently, our formalization can be used to implement materialization in various systems. Although our formalization is more suitable for implementation systems that support metaclasses, it can be also used to assist an implementation in non metaclass-based systems. Furthermore, our approach can be followed to formalize other new generic relationships. This work had also demonstrated the power of the TELOS language to express the requirements related to the materialization semantics. To fully formalize materialization, we proposed two suggestions to extend TELOS. One consisted of the definition of a new predicate required to formalize the cardinality [Cmin , Cmax ]. The second one consisted of the extension of the deductive rule definition for specifying some operations we would wish to be automatically executed by the system when given preconditions are satisfied. Our work has several continuations. An interesting one deals with the formalization of the common semantics of a large repository of new generic relationships (e.g., Member-Of [MPS95], Role-Of [WDJS94], Part-Of [KP97], Ownership [HPYG95]). Another continuation will be to start from our formalization and try to realize an effective implementation in a target system (e.g., ConceptBase) in such manner that it will be possible to evaluate this system with regard to its power in the implementation of materialization semantics. Acknowledgements. I wish to thank all the colleagues in the YEROOS project for their critical comments on an earlier draft of this article and particularly M. Guy Fokou. I am also grateful to Christoph Quix (RWTH Technical University of Aachen) for many useful discussions about TELOS and ConceptBase. Finally, I thank Professor Matthias Jarke (RWTH Technical University of Aachen) for the permission to use the ConceptBase system.
References [Dah97]
M. Dahchour. Formalizing materialization in the TELOS data model. Technical Report TR-97/28, IAG-QANT, Universit´e catholique de Louvain, Belgium, November 1997. [DPZ96] M. Dahchour, A. Pirotte, and E. Zim´ anyi. Metaclass implementation of materialization. Technical Report YEROOS TR-96/06, January 1996. Submitted for publication. [DPZ97] M. Dahchour, A. Pirotte, and E. Zim´ anyi. Metaclass implementations of generic relationships. Technical Report YEROOS TR-97/25, 1997. Submitted for publication. [GSR96] G. Gottlob, M. Schrefl, and B. R¨ ock. Extending object-oriented systems with roles. ACM Trans. on Office Information Systems, 14(3):268–296, 1996. [HGPK94] M. Halper, J. Geller, Y. Perl, and W. Klas. Integrating a part relationship into an open OODB system using metaclasses. In N.R. Adam, B.K. Bhargava, and Y. Yesha, editors, Proc. of the 3rd Int. Conf. on Information and Knowledge Management, CIKM’94, pages 10–17, Gaithersburg, Maryland, November 1994. ACM Press.
420
Mohamed Dahchour
[HPYG95] M. Halper, Y. Perl, O. Yang, and J. Geller. Modeling business applications with the OODB ownership Relationship. In Proc. of the 3rd Int. Conf. on AI Applications on Wall St., pages 2–10, June 1995. [JJS96] M. Jarke, M.A. Jeusfeld, and M. Staudt. ConceptBase V4.1 User Manual. 1996. [KFP88] G. E. Kaiser, P. H. Feiler, and S. S. Popovich. Intelligent assistance for software development and maintenance. IEEE Software, 5(3):40–49, 1988. [KP97] M. Kolp and A. Pirotte. An aggregation model and its C++ implementation. In Proc. of the 4th Int. Conf. on Object-Oriented Information Systems, Brisbane, Australia, 1997. To appear. [KS95] W. Klas and M. Schrefl. Metaclasses and their application. LNCS 943. Springer-Verlag, 1995. [Lie86] H. Lieberman. Using prototypical objects to implement shared behavior in object oriented systems. In N.K. Meyrowitz, editor, Proc. of the Conf. on Object-Oriented Programming Systems, Languages and Applications, OOPSLA’86, pages 214–223, Portland, Oregon, November 1986. ACM SIGPLAN Notices 21(11), 1986. [MBJK90] J. Mylopoulos, A. Borgida, M. Jarke, and M. Koubarakis. Telos: representing knowledge about informations systems. ACM Trans. on Office Information Systems, 8(4):325–362, 1990. [MP93] R. Motschnig-Pitrik. The semantics of parts versus aggregates in data/knowledge modelling. In C. Rolland, F. Bodart, and C. Cauvet, editors, Proc. of the 5th Int. Conf. on Advanced Information Systems Engineering, CAiSE’93, LNCS 685, pages 352–373, Paris, France, June 1993. Springer-Verlag. [MPK96] R. Motschnig-Pitrik and J. Kaasboll. Part-whole relationship categories and their application in object-oriented analysis. In Proc. of the 5th Int. Conf. on Information System Development, ISD’96, September 1996. [MPS95] R. Motschnig-Pitrik and V.C. Storey. Modelling of set membership: The notion and the issues. Data & Knowledge Engineering, 16(2):147–185, 1995. [PZMY94] A. Pirotte, E. Zim´ anyi, D. Massart, and T. Yakusheva. Materialization: a powerful and ubiquitous abstraction pattern. In J. Bocca, M. Jarke, and C. Zaniolo, editors, Proc. of the 20th Int. Conf. on Very Large Databases, VLDB’94, pages 630–641, Santiago, Chile, 1994. Morgan Kaufmann. [WDJS94] R. Wieringa, W. De Jonge, and P. Spruit. Roles and dynamic subclasses: a modal logic approach. In M. Tokoro and R. Pareschi, editors, Proc. of the 8th European Conf. on Object-Oriented Programming, ECOOP’94, LNCS 821, pages 32–59, Bologna, Italy, July 1994. Springer-Verlag.
