Lecture Notes in Computer Science Edited by G. Goos, J. Hartmanis, and J. van Leeuwen
2453
3
Berlin Heidelberg New York Barcelona Hong Kong London Milan Paris Tokyo
Abdelkader Hameurlain Rosine Cicchetti Roland Traunmüller (Eds.)
Database and Expert Systems Applications 13th International Conference, DEXA 2002 Aix-en-Provence, France, September 2-6, 2002 Proceedings
13
Series Editors Gerhard Goos, Karlsruhe University, Germany Juris Hartmanis, Cornell University, NY, USA Jan van Leeuwen, Utrecht University, The Netherlands Volume Editors Abdelkader Hameurlain Université Paul Sabatier, IRIT 118 route de Narbonne, 31062 Toulouse Cedex, France E-mail:
[email protected] Rosine Cicchetti Université Aix-Marseille II, IUT, Département Informatique 413 Avenue Gaston Berger, 13625 Aix-en-Provence Cedex 1, France E-mail:
[email protected] Roland Traunmüller University of Linz, Institute of Applied Computer Science Altenbergerstr. 69, 4040 Linz, Austria E-mail:
[email protected] Cataloging-in-Publication Data applied for Die Deutsche Bibliothek - CIP-Einheitsaufnahme Database and expert systems applications : 13th international conference ; proceedings / DEXA 2002, Aix-en-Provence, France, September 2 - 6, 2002. Abdelkader Hameurlain ... (ed.). - Berlin ; Heidelberg ; New York ; Barcelona ; Hong Kong ; London ; Milan ; Paris ; Tokyo : Springer, 2002 (Lecture notes in computer science ; Vol. 2453) ISBN 3-540-44126-3
CR Subject Classification (1998): H.2, H.3, I.2.1, H.4, H.5, J.1 ISSN 0302-9743 ISBN 3-540-44126-3 Springer-Verlag Berlin Heidelberg New York This work is subject to copyright. All rights are reserved, whether the whole or part of the material is concerned, specifically the rights of translation, reprinting, re-use of illustrations, recitation, broadcasting, reproduction on microfilms or in any other way, and storage in data banks. Duplication of this publication or parts thereof is permitted only under the provisions of the German Copyright Law of September 9, 1965, in its current version, and permission for use must always be obtained from Springer-Verlag. Violations are liable for prosecution under the German Copyright Law. Springer-Verlag Berlin Heidelberg New York a member of BertelsmannSpringer Science+Business Media GmbH http://www.springer.de © Springer-Verlag Berlin Heidelberg 2002 Printed in Germany Typesetting: Camera-ready by author, data conversion by Steingräber Satztechnik GmbH, Heidelberg Printed on acid-free paper SPIN: 10871136 06/3142 543210
Preface 2002 DEXA, the 13th International Conference on Database and Expert Systems Applications was held on September 2–6, 2002, at the Université Aix–Marseille II, France. The quickly growing field of information systems required the establishment of more specialized discussion platforms (the DaWaK conference, EC-Web conference, eGOV conference and DEXA workshops), and there were held in parallel with DEXA, also in Aix-en-Provence. The resulting book was prepared with great effort. Starting with the preparation of submitted papers, the papers went through the reviewing process. The accepted papers were revised to final versions by their authors and arranged to the conference program. This year 241 papers were submitted and our thanks go to all who have contributed. The program committee and the supporting reviewers produced altogether about 730 referee reports, on average three reports per paper, and selected 89 papers for presentation. The papers presented here encompass the extensive domain of databases; together with the other conferences and workshops of the DEXA event cluster a vast part of applied computer science was covered. In this way DEXA has blazed the trail. At this point we would like to acknowledge to all institutions which actively supported this conference and made it possible. These are: • IUT (Université Aix – Marseille II), • FAW, • DEXA Association, • the Austrian Computer Society, • and Microsoft Research A conference like DEXA would not be possible without the dedicated work of several people in the background. So we want to thank the whole program committee for the thorough refereeing process. We specially appreciated the essential commitment of Gabriela Wagner (FAW, University of Linz). September 2002
Rosine Cicchetti (IUT, Université Aix–Marseille II) Abdelkader Hameurlain (IRIT, Université Paul Sabatier, France) Roland Traunmüller (University of Linz)
Organization
Program Committee General Chairperson Abdelkader Hameurlain, IRIT, Université Paul Sabatier, France
Conference Program Chairpersons Lotfi Lakhal, University of Marseille, France Roland Traunmüller, University of Linz, Austria
Workshop Chairpersons A Min Tjoa, Technical University of Vienna, Austria Roland R. Wagner, FAW, University of Linz, Austria
Publication Chairperson Vladimir Marik, Czech Technical University, Czech Republic
Program Committee Members Michel Adiba, IMAG - Laboratoire LSR, France Hamideh Afsarmanesh, University of Amsterdam, The Netherlands Jens Albrecht, Oracle GmbH, Germany Ala Al-Zobaidie, University of Greenwich, UK Bernd Amann, CNAM, France Frederic Andres, NII, Japan Kurt Bauknecht, University of Zurich, Switzerland Trevor Bench-Capon, University of Liverpool, United Kingdom Alfs Berztiss, University of Pittsburgh, USA Sourav S Bhowmick, Nanyang Technological Univ., Singapore Jon Bing, University of Oslo, Norway Omran Bukhres, Purdue University, USA Luis Camarinah-Matos, New University of Lisbon, Portugal Antonio Cammelli, IDG-CNR, Italy Wojciech Cellary, University of Economics at Poznan, Poland Stavros Christodoulakis, Technical University of Crete, Greece Panos Chrysanthis, Univ. of Pittsburgh & Carnegie Mellon Univ., USA Paolo Ciaccia, University of Bologna, Italy Rosine Cicchetti, University of Marseille, France Christine Collet, LSR-IMAG, France Carlo Combi, University of Verona, Italy William Bruce Croft, University of Massachusetts, USA John Debenham, University of Technology, Sydney, Australia Misbah Deen, University of Keele, United Kingdom Nina Edelweiss, University of Rio Grande do Sul, Brazil Johann Eder, University of Klagenfurt, Austria Thomas Eiter, Technical University of Vienna, Austria Gregor Engels, University of Paderborn, Germany
VII
VIII
Organization
Peter Fankhauser, GMD-IPSI, Germany Eduardo Fernandez, Florida Atlantic University, USA Simon Field, IBM Research Division, Switzerland Burkhard Freitag, University of Passau, Germany Peter Funk, Mälardalen University, Sweden Antonio L. Furtado, University of Rio de Janeiro, Brazil Georges Gardarin, University of Versailles, France Parke Godfrey, The College of William and Mary, Canada Paul Grefen, University of Twente, The Netherlands William Grosky, University of Michigan, USA Igor T. Hawryszkiewycz, University of Technology, Sydney, Australia Mohamed Ibrahim, University of Greenwich, UK Yahiko Kambayashi, Kyoto University, Japan Magdi N. Kamel, Naval Postgraduate School, USA Nabil Kamel, American University in Cairo, Egypt Gerti Kappel, University of Linz, Austria Kamalar Karlapalem, University of Science and Technology, Hong Kong Randi Karlsen, University of Tromsö, Norway Rudolf Keller, Zühlke Engineering AG, Switzerland Myoung Ho Kim, KAIST, Korea Masaru Kitsuregawa, Tokyo University, Japan Gary J. Koehler, University of Florida, USA Donald Kossmann, Technical University of Munich, Germany Jaques Kouloumdjian, INSA, France Petr Kroha, Technical University Chemnitz-Zwickau, Germany Josef Küng, University of Linz, Austria Jiri Lazansky, Czech Technical University, Czech Republic Michel Leonard, University of Geneve, Switzerland Tok Wang Ling, National University of Singapore, Singapore Mengchi Liu, University of Regina, Canada Fred Lochovsky, Hong Kong Univ. of Science & Technology, Hong Kong Peri Loucopoulos, UMIST, United Kingdom Sanjai Kumar Madria, University of Missouri-Rolla, USA Akifumi Makinouchi, Kyushu University, Japan Heinrich C. Mayr, University of Klagenfurt, Austria Vladimir Marik, Czech Technical University, Czech Republic Simone Marinai, University of Florence, Italy Subhasish Mazumdar, New Mexico Tech, USA Robert Meersman, Free University Brussels, Belgium Elisabeth Metais, CEDRIC/CNAM of Paris, France Mukesh Mohania, IBM-IRL, India Tadeusz Morzy, Poznan University of Technology, Poland Noureddine Mouaddib, University of Nantes, France Günter Müller, University of Freiburg, Germany Erich J. Neuhold, Fraunhofer-IPSI, Germany Gultekin Ozsoyoglu, University Case Western Research, USA Georgios Pangalos, University of Thessaloniki, Greece
Organization
Stott Parker, University of Los Angeles (UCLA), USA Oscar Pastor, Technical University of Valencia, Spain Marco Patella, University of Bologna, Italy Barbara Pernici, Politecnico di Milano, Italy Günter Pernul, University of Essen, Germany Gerald Quirchmayr, University of Vienna, Austria Fausto Rabitti, CNUCE-CNR, Italy Isidro Ramos, Technical University of Valencia, Spain Harald Reiterer, University of Konstanz, Germany Norman Revell, Middlesex University, UK Sally Rice, Unversity of South Australia, Australia John Roddick, Flinders University of South Australia, Australia Colette Rolland, University Paris I, Sorbonne, France Elke Rundensteiner, Worcester Polytechnic Institute, USA Domenico Sacca, University of Calabria, Italy Marinette Savonnet, University of Bourgogne, France Erich Schweighofer, University of Vienna, Austria Michael H. Smith, University of California, USA Giovanni Soda, University of Florence, Italy Harald Sonnberger, EUROSTAT, Luxembourg Günther Specht, University of Ulm, Germany Uma Srinivasan, CSIRO, Australia Bala Srinivasan, Monash University, Australia Olga Stepankova, Czech Technical University, Czech Republic Zbigniew Struzik, CWI, Amsterdam, The Netherlands Makoto Takizawa, Tokyo Denki University, Japan Katsumi Tanaka, Kyoto University, Japan Zahir Tari, University of Melbourne, Australia Stephanie Teufel, University of Fribourg, Germany Jukka Teuhola, University of Turku, Finland Bernd Thalheim, Technical University of Cottbus, Germany Jean Marc Thevenin, University of Toulouse, France Helmut Thoma, IBM Global Services Basel, Switzerland A Min Tjoa, Technical University of Vienna, Austria Aphrodite Tsalgatidou, University of Athens, Greece Susan Urban, Arizona State University, USA Krishnamurthy Vidyasankar, Memorial Univ. of Newfoundland, Canada Pavel Vogel, Technical University of Munich, Germany Roland R. Wagner, University of Linz, Austria Michael Wing, Middlesex University, UK Werner Winiwarter, EC3, Austria Vilas Wuwongse, Asian Institute of Technology, Thailand Gian Piero Zarri, CNRS, France Arkady Zaslavsky, Monash University, Australia Aidong Zhang, State Univ. of New York at Buffalo, USA
IX
X
Organization
External Reviewers
Vassilios Stoumpos Dimitrios Theotokis Mara Nikolaidou Thomi Pilioura Olli Nevalainen Stéphane Grandcolas Mohand-Said Hacid Noel Novelli Werner Retschitzegger Wieland Schwinger Morad Benyoucef Angela Bonifati Michael Hanus Carlo Meghini Pasquale Savino Giuseppe Amato Claudio Gennaro Gillian Dobbie Eng Koon Sze Maria Indrawan Campbell Wilson Manish Bhide Upendra Sharma Andreas Goerdt Marco Guenther Kaiyang Liu Jie Mei Joshua Pun Jiying Wang Jose Martinez Mohammed Quafafou Masato Oguchi Masatoshi Arikawa Tadashi Ohmori
Haruo Yokota Miyuki Nakano Krishina Reddy Xiaofang Zhou Harald Kosch Christian Koncilia Wolfgang Gruber Elisabetta Grazzini Andrea Passerini Alessandro Vullo Cesar Garita Ammar Benabdelkader Ersin Kaletas Anne Frenkel Y.D. Chung J.H. Son J.H. Kim Maciej Zakrzewicz Marek Wojciechowski Zbyszko Krolikowski Matilde Celma José Ramón Mendivil Pedro Sánchez Palma Fredj Dridi Holger Grabow Markus Hütten Torsten Priebe Torsten Schlichting Andreas Gruenhagen Steffen Rost Genoveva Vargas Solar J. L. Zechinelli-Martini Tuyet-Trinh Vu Cyril Labbe
Jérome Gensel Khalid Belhajjame Yimin Wu Yong Shi Vicente Pelechano Emilio Insfran Juan Sanchez Beatrice Finance Fabio Porto Sean W. Siqueira S. Chatvichienchai Akira Okada Predrag Knezevic Rodolfo Stecher Thomas Risse Wendy Wu Tony Stockman Wojciech Wiza Krzysztof Walczak Franck Morvan Carina Dorneles Fábio Zschornack Mirella Moura Moro R. De Matos Galante Sandro Da Silva Camargo Andreas Erat Thomas Schlienger Martin Steinert Thomas Christoffel Wijnand Derks Rik Eshuis Djoerd Hiemstra
Table of Contents
Invited Talk Issues in Monitoring Web Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . S. Abiteboul, INRIA and Xyleme SA, France
1
Web I Controlled Caching of Dynamic WWW Pages . . . . . . . . . . . . . . . . . . . . . . . . . . . . C. Vassilakis, G. Lepouras, Greece
9
Deriving and Verifying Statistical Distribution of a Hyperlink-Based Web Page Quality Metric . . . . . . . . . . . . . . . . . . . . . . . . . . . 19 D. Dhyani, S.S. Bhowmick, W.K. Ng, Singapore MWM: Retrieval and Automatic Presentation of Manual Data for Mobile Terminals . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29 M. Shikata , A. Nadamoto , K. Sumiya , K. Tanaka , Japan Ensuring Recovery for SACReD Web Transactions in the E-commerce Applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40 M. Younas , B. Eaglestone , United Kingdom
Workflow Workflow Simulation across Multiple Workflow Domains . . . . . . . . . . . . . . . . . . . 50 X. Zhao, China Transactional Workflows or Workflow Transactions? . . . . . . . . . . . . . . . . . . . . . . . 60 P. Grefen, The Netherlands Collaboration Flow Management: A New Paradigm for Virtual Team Support . . . . 70 J. Lonchamp, France Domain Knowledge-Based Automatic Workflow Generation . . . . . . . . . . . . . . . . . 81 S.A. Chun, V. Atluri, N.R. Adam, USA
Web II A Comparison of Alternative Encoding Mechanisms for Web Services . . . . . . . . . 93 M. Cai, S. Ghandeharizadeh, R. Schmidt, S. Song, USA Capturing Semantics in HTML Documents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103 M. Liu, Canada
XII
Table of Contents
Data Warehouses and Datamining I Assessment of Spatial Data Mining Tools for Integration into an Object-Oriented GIS (GIDB) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113 R. Ladner, F.E. Petry, USA A Model and a Toolkit for Supporting Incremental Data Warehouse Construction . . . . . . . . . . . . . . . . . . . 123 P. Naggar , L. Pontieri , M. Pupo , G. Terracina , E. Virardi, Italy Efficient Automated Mining of Fuzzy Association Rules . . . . . . . . . . . . . . . . . . . . 133 M. Kaya , R. Alhajj , F. Polat , A. Arslan, Canada, Turkey
Applications I A Two-Phase Stock Trading System Using Distributional Differences . . . . . . . . . . 143 S.-D. Kim , J.W. Lee , L. Lee, J. Chae, Korea A Multi-agent Q-learning Framework for Optimizing Stock Trading Systems . . . . 153 J.W. Lee , J. O, Korea Using Adaptive Programming in Legal Advisory Systems . . . . . . . . . . . . . . . . . . . 163 P. Kroha, S. B¨urgel, Germany Web Information Retrieval Based on the Localness Degree . . . . . . . . . . . . . . . . . . . 172 C. Matsumoto , Q. Ma , K. Tanaka, Japan
Data Warehouses and Datamining II Run-Time Load Balancing System on SAN-connected PC Cluster for Dynamic Injection of CPU and Disk Resource – A Case Study of Data Mining Application – . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 182 K. Goda , T. Tamura , M. Oguchi, M. Kitsuregawa, Japan Variable Sized Partitions for Range Query Algorithms . . . . . . . . . . . . . . . . . . . . . . 193 T.W. Ling, W.C. Low, Z.W. Luo, S.Y. Lee, H.G. Li, Singapore A Comparison of Data Warehouse Development Methodologies Case Study of the Process Warehouse . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 203 B. List , R.M. Bruckner , K. Machaczek , J. Schiefer, Austria, USA Self-maintainable Data Warehouse Views Using Differential Files . . . . . . . . . . . . 216 L. Wookey , Y. Hwang , S.-H. Kang , S. Kim , C. Kim , Y. Lee, Korea
Applications II Using ECA Rules in Database Systems to Support Clinical Protocols . . . . . . . . . . 226 K. Dube , B. Wu , J.B. Grimson, Ireland
Table of Contents
XIII
Establishment of Virtual Enterprise Contracts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 236 G. Quirchmayr , Z. Milosevic , R. Tagg , J. Cole , S. Kulkarni, Austria, Australia Multiple Regression Analysis in Crime Pattern Warehouse for Decision Support . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 249 D. Dzemydiene , V. Rudzkiene, Lithuania A Practical Walkthrough of the Ontology Derivation Rules . . . . . . . . . . . . . . . . . . 259 C. Wouters , T. Dillon , W. Rahayu , E. Chang, Australia
XML What’s Hard about XML Schema Constraints? . . . . . . . . . . . . . . . . . . . . . . . . . . . . 269 M. Arenas , W. Fan , L. Libkin, Canada Application of rUID in Processing XML Queries on Structure and Keyword . . . . . 279 D.D. Kha , M. Yoshikawa , S. Uemura, Japan Translating Access Authorizations for Transformed XML Documents . . . . . . . . . . 290 S. Chatvichienchai, M. Iwaihara, Y. Kambayashi, Japan XRL: A XML-Based Query Language for Advanced Services in Digital Libraries . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 300 J.M. P´erez, M.J. Aramburu, R. Berlanga, Spain
Applications III Norma-System: A Legal Document System for Managing Consolidated Acts . . . . 310 M. Palmirani, R. Brighi, Italy Electricity Market Price Forecasting: Neural Networks versus Weighted-Distance k Nearest Neighbours . . . . . . . . . . . . . . . . . . . . . . . . . . . 321 A. Troncoso Lora , Jos´e Riquelme Santos , Jes´us Riquelme Santos , J.L. Mart´ınez Ramos , A. G´omez Exp´osito, Spain Temporal Pattern Mining of Moving Objects for Location-Based Service . . . . . . . 331 J.D. Chung , O.H. Paek , J.W. Lee, K.H. Ryu, Korea
Distributed Systems I MEDWRAP: Consistent View Maintenance over Distributed Multi-relation Sources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 341 A.S. Varde, E.A. Rundensteiner, USA Estimating Costs of Path Expression Evaluation in Distributed Object Databases . 351 G. Ruberg, F. Bai˜ao, M. Mattoso, Brazil Profile Driven Data Management for Pervasive Environments . . . . . . . . . . . . . . . . 361 F. Perich, S. Avancha, D. Chakraborty, A. Joshi, Y. Yesha, USA
XIV
Table of Contents
Applications IV INVISIP: Usage of Information Visualization Techniques to Access Geospatial Data Archives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 371 S. G¨obel , J. Haist , H. Reiterer , F. M¨uller, Germany Eureka! : A Tool for Interactive Knowledge Discovery . . . . . . . . . . . . . . . . . . . . . . 381 G. Manco , C. Pizzuti , D. Talia, Italy Electronic Tickets on Contactless Smartcard Database . . . . . . . . . . . . . . . . . . . . . . 392 K. Kuramitsu, K. Sakamura, Japan
Distributed Systems II Mobile Agents for Distributed Transactions of a Distributed Heterogeneous Database System . . . . . . . . . . . . . . . . . . . . . . . . . . 403 D.Y. Ye, M.C. Lee, T.I. Wang, Taiwan An Adaptive Direction-Based Location Update Scheme for Next Generation PCS Networks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 413 D.-X Ou , K.-Y. Lam , D.-C. Dong, China, Hong Kong ADEES: An Adaptable and Extensible Event Based Infrastructure . . . . . . . . . . . . . 423 G. Vargas-Solar, C. Collet, France
Knowledge Engineering I Using J-Pruning to Reduce Overfitting of Classification Rules in Noisy Domains . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 433 M. Bramer, United Kingdom A Framework for Reasoning on Objects with Geometric Constraints . . . . . . . . . . . 443 H. Hajji , E. Terzi, France, USA Deriving Abstract Views of Multi-granularity Temporal Constraint Networks . . . . . . . . . . . . . . . . . . . . . . . . 454 C. Bettini, S. Ruffini, Italy Representing Constraint Business Rules Extracted from Legacy Systems . . . . . . . 464 G. Fu , J. Shao , S.M. Embury , W.A. Gray, United Kingdom
Knowledge Engineering II Agents for Industry Process Management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 474 J. Debenham, Australia A New Efficient Data Cleansing Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 484 L. Zhao, S.S. Yuan, S. Peng, L.T. Wang, Singapore
Table of Contents
XV
Promoting Recommendations: An Attack on Collaborative Filtering . . . . . . . . . . . 494 M.P. O’Mahony, N.J. Hurley, G.C.M. Silvestre, Ireland
Advanced Databases I Bringing Together Description Logics and Database in an Object Oriented Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 504 M. Roger, A. Simonet, M. Simonet, France Objects and Roles in the Stack-Based Approach . . . . . . . . . . . . . . . . . . . . . . . . . . . 514 A. Jod áowski , P. Habela , J. P áodzie´n , K. Subieta, Poland Dynamic Schema Evolution Management Using Version in Temporal Object-Oriented Databases . . . . . . . . . . . . . . . . . . . . . 524 R. de Matos Galante, A. Bueno da Silva Roma, A. Jantsch, N. Edelweiss, C. Saraiva dos Santos, Brazil
Invited Talk Efficient Data Mining Based on Formal Concept Analysis . . . . . . . . . . . . . . . . . . . 534 G. Stumme, Germany
Advanced Databases II Two Approaches to Event Definition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 547 A. Galton , J.C. Augusto, United Kingdom CoSMo: An Approach Towards Conceptual Security Modeling . . . . . . . . . . . . . . . 557 C. Artelsmair , W. Essmayr , P. Lang, R. Wagner , E. Weippl, Austria Exploring the Concept of Group Interaction through Action in a Mobile Context . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 567 R. Harr, Sweden Inter-system Triggering for Heterogeneous Database Systems . . . . . . . . . . . . . . . . 577 W. Guan, R.K. Wong, Australia
Query I Constructing Inter-relational Rules for Semantic Query Optimisation . . . . . . . . . . 587 B.G.T. Lowden, J. Robinson, United Kingdom About Selections and Joins in Possibilistic Queries Addressed to Possibilistic Databases . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 597 P. Bosc , L. Duval , O. Pivert, France LeedsCQ: A Scalable Continual Queries System . . . . . . . . . . . . . . . . . . . . . . . . . . . 607 S. Khan, P.L. Mott, United Kingdom
XVI
Table of Contents
TVQL - Temporal Versioned Query Language . . . . . . . . . . . . . . . . . . . . . . . . . . . . 618 M. Moura Moro , N. Edelweiss , A. Pereira Zaupa, C. Saraiva dos Santos, Brazil
Advanced Databases III Approach and Model for Business Components Specification . . . . . . . . . . . . . . . . . 628 P. Ramadour, C. Cauvet, France
Query II The TrailTRECer Framework – A Platform for Trail-Enabled Recommender Applications . . . . . . . . . . . . . . . . . . 638 E. Gams, T. Berka, S. Reich, Austria Adaptive Method for Range Top-k Queries in OLAP Data Cubes . . . . . . . . . . . . . 648 Z.X. Loh, T.W. Ling, C.H. Ang, S.Y. Lee, Singapore dbRouter – A Scaleable and Distributed Query Optimization and Processing Framework . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 658 W.H. Tok, S. Bressan, Singapore Optimal Parallel I/O for Range Queries through Replication . . . . . . . . . . . . . . . . . 669 K. Frikken , M. Atallah , S. Prabhakar, R. Safavi-Naini, USA, Australia
Advanced Databases IV A Database for Repetitive, Unpredictable Moving Objects . . . . . . . . . . . . . . . . . . . 679 P. Werstein, J.R. McDonald, New Zealand Hierarchical Storage Support and Management for Large-Scale Multidimensional Array Database Management Systems . . . . . . . 689 B. Reiner , K. Hahn , G. H¨ofling , P. Baumann, Germany Multiple Visual Representation of Temporal Data . . . . . . . . . . . . . . . . . . . . . . . . . . 701 C. Daassi , M.-C. Fauvet , L. Nigay, France
Query III Query Optimization via Empty Joins . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 710 J. Gryz, D. Liang, Canada Efficient Processing of XPath Queries Using Indexes . . . . . . . . . . . . . . . . . . . . . . . 721 Y. Chen S. Madria , K. Passi , S. Bhowmick, USA, Canada, Singapore Efficient Evaluation of Continuous Range Queries on Moving Objects . . . . . . . . . 731 D. Kalashnikov, S. Prabhakar, S. Hambrusch, W. Aref, USA Partition-Based Similarity Join in High Dimensional Data Spaces . . . . . . . . . . . . . 741 H. Shin , B. Moon , S. Lee, Korea, USA
Table of Contents
XVII
Invited Talk Heterogeneous Data Source Integration and Evolution . . . . . . . . . . . . . . . . . . . . . . 751 M. Bouzeghoub , B. Farias L´oscio , Z. Kedad , A. Soukane, France, Brazil
Information Retrieval I Information Retrieval System for XML Documents . . . . . . . . . . . . . . . . . . . . . . . . 758 K. Hatano, H. Kinutani , M. Yoshikawa , S. Uemura, Japan A System for Retrieval and Digest Creation of Video Data Based on Geographic Objects . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 768 T. Ueda , T. Amagasa, M. Yoshikawa , S. Uemura, Japan Similarity Image Retrieval System Using Hierarchical Classification . . . . . . . . . . . 779 M. Tada , T. Kato , I. Shinohara, Japan Information Extraction – Tree Alignment Approach to Pattern Discovery in Web Documents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 789 A. Hemnani, S. Bressan, Singapore
Query IV Processing of Continuous Queries over Unlimited Data Streams . . . . . . . . . . . . . . 799 E. Vossough , J.R. Getta, Australia ISA - An Incremental Hyper-sphere Approach for Efficiently Solving Complex Vague Queries . . . . . . . . . . . . . . . . . . . . . . . . . . . . 810 T.K. Dang, J. K¨ung, R. Wagner, Austria Parallel Query Support for Multidimensional Data: Inter-object Parallelism . . . . . 820 K. Hahn , B. Reiner , G. H¨ofling , P. Baumann, Germany
Information Retrieval II Pruning Training Corpus to Speedup Text Classification . . . . . . . . . . . . . . . . . . . . 831 J. Guan , S. Zhou, China Towards Retrieval of Visual Information Based on the Semantic Models . . . . . . . . 841 Y. Park, P. Kim , W. Kim, J.-J. Song, S. Panchanathan, USA, Korea 3D Object Retrieval by Shape Similarity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 851 J.-J. Song, F. Golshani, USA
Indexing I Binary-Tree Histograms with Tree Indices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 861 F. Buccafurri , F. Furfaro , G. Lax, D. Sacc`a, Italy Indexing Genomic Databases for Fast Homology Searching . . . . . . . . . . . . . . . . . . 871 T.-H. Ong , K.-L. Tan , H. Wang, Singapore
XVIII
Table of Contents
Bitmap Indices for Speeding Up High-Dimensional Data Analysis . . . . . . . . . . . . 881 K. Stockinger, Switzerland and Austria
Information Retrieval III A Method of Improving Feature Vector for Web Pages Reflecting the Contents of Their Out-Linked Pages . . . . . . . . . . . . . . . . . . . . . . . . . 891 K. Sugiyama , K. Hatano , M. Yoshikawa, S. Uemura, Japan On Combining Link and Contents Information for Web Page Clustering . . . . . . . . 902 Y. Wang, M. Kitsuregawa, Japan
Indexing II MKL-Tree: A Hierarchical Data Structure for Indexing Multidimensional Data . . 914 R. Cappelli, A. Lumini, D. Maio, Italy Boolean Bounding Predicates for Spatial Access Methods . . . . . . . . . . . . . . . . . . . 925 M. Thomas, J.M. Hellerstein, USA Clustered Indexing Technique for Multidimensional Index Structures . . . . . . . . . . 935 G.-H. Cha, Y.-I. Yoon, Korea Database and Expert Systems 2002 “ Quo vadis”? . . . . . . . . . . . . . . . . . . . . . . . . . . 945 A. M. Tjoa , R.R. Wagner, Austria Author Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 949
Issues in Monitoring Web Data Serge Abiteboul INRIA and Xyleme
Abstract. The web is turning from a collection of static documents to a global source of dynamic knowledge. First, HTML is increasingly complemented by the more structured XML format augmented with mechanisms for providing semantics such as RDF. Also, we believe that the very passive vision of web sites promoted by the current HTTP will soon turn into a much more active one based on pub/sub and web services. Thus, the monitoring of the web is likely to raise new challenges for many years. We consider here some applications involving web monitoring and some issues they raise.
1
Introduction
The web constitutes the largest body of universally accessible information throughout the history of humanity and keeps growing at a healthy pace. This new frontier that was first developed and used by a small number of experts is now turning into a huge distributed information base available to the public at large. However, besides its size, distribution and heterogeneity, the web is difficult to use because in some respect, it is also a huge, evolving junk yard: – A lot of the information cannot be trusted, e.g., spamming, rumors, etc. – a lot of the information on the web is stale, e.g., dead web sites or pages no longer maintained. – even when a site or a page is alive, our vision of it may be out-to-date, e.g., full-text indexes on the web are often not aware of recent changes of pages. To overcome this difficulty, we need to “monitor” the web. More precisely, web robots need to permanently look for new information on the web, typically by crawling it or by querying databases published on the web. Robots then need to analyze this information, e.g., to evaluate the quality of data that has been found and classify it. Finally, they need to monitor (in the sense of performing some surveillance) the data in order to detect changes. We first consider the nature of the data one may wish to monitor on the web. We then look at typical applications involving web monitoring. While doing so, we consider technical issues raised by web monitoring. The present paper is motivated by recent works of the author. In particular, it has been motivated by works on the creation of a warehouse of XML data found on the web [11] that lead to the creation of a start-up named Xyleme [20]. The present paper presents a biased vision of a very diverse and active field and is not in any sense a comprehensive survey. R. Cicchetti et al. (Eds.): DEXA 2002, LNCS 2453, pp. 1–8, 2002. c Springer-Verlag Berlin Heidelberg 2002
2
S. Abiteboul
2
The Nature of Data to Monitor
The web was first developed very homogeneously around HTML. In other words, the bulk of the web consisted of hypertext documents. While maturing, the web became increasingly complex. Nowadays, the following information may be found: – HTML still forms the backbone of the web, but popular document formats such as DOC, PDF, PS are massively present. – A new web exchange format, namely XML [19], promoted by the w3c [21] and the industry, is radically changing the web. XML marries, in some sense, the document world with the database world [1]. XML can be used to represent documents, but also structured or semistructured data [1]. – A lot of web data is typically retrieved as a result of queries, in particular queries to database systems. Typically, this means that data (query parameters) has to be fed to obtain more data. The term deep web or hidden web [10] are used for such accesses typically via forms, scripts or using a new standard for web services, SOAP [15]. – Finally, multimedia formats for images, sound, video are crucial components of the web that we will not discuss here. Note that web robots mostly reach the HTML portion of the web although some may index other formats (e.g., PDF for Google [13]) or crawl XML as well (e.g. Xyleme [20]). Databases are typically seen as stop points by crawlers. So, the deep web (larger than the surface web) is still out of the reach of crawlers. However, most people would probably agree that they use more regularly the deep web (e.g., for ordering tickets, or when querying yellow pages) than the surface web. With the web maturing, we believe that this will be increasingly the case. When considering the nature of data to monitor, one should also mention: – Public vs. private web: the private web is typically accessible on an Intranet or on the Internet protected by passwords. The main issue here is the control of access rights. From a technical perspective, there is no real difference between private and public data. – Static vs. dynamic. One can imagine a static page that is kept very fresh by 24-hour operators and similarly a dynamic page that for years will return the same value. The main issues are those already mentioned: the format of the data and whether or not there is a need to provide query parameters.
3
Difference with Traditional Databases
Although XML is often presented as a new model for data storage with its query language (Xquery [18]), we prefer to view XML here essentially as an exchange format ignoring the nature of the storage (relational, native or other) or the way the data has been produced. Indeed, in this spirit, there is more and
Issues in Monitoring Web Data
3
more interest in “stream” queries for XML data, see, e.g., [3]. By this we mean that the query refers to data that is being received vs. data that is stored in a database. Note that the problem is intrinsically different. The core of database technology consists in secondary storage management and in taking advantage of indexes. Stream queries are typically applied to main memory data and indexes are not available a-priori. Data management in a web context presents other particularities that we mention next. First, web data is by nature distributed and heterogeneous. This is in the spirit of distributed databases or more precisely in the spirit of federated databases that stress the independence of data sources that are components of such systems. A main new aspect when considering, for instance, data integration on the web, is the number of sources (tens, hundreds, thousands). Moreover, federated databases typically consider environments where the semantics of data in sources of interest is well understood. This assumption does not scale to the web. We have to be able to discover new sources dynamically and analyze them to understand their semantics on the fly. This clearly complicates monitoring that now has to include a significant artificial intelligence component. Last but not least, another difference (say, to federated databases) is that the web does not provide for change control on data we are interested in. For HTML pages, it is possible to register with services providing notifications when pages of interest change. This is clearly a very primitive form of change control compared to the sophisticate triggering mechanisms supported by traditional database systems. One can measure the limitations of such approaches when one knows that this is typically implemented by regularly (once a day, a week, a month, more?) reading the page to check whether it has changed and inspecting some hash value of the text of the page. In the remainder of the paper, we consider applications that deal with web monitoring and mention some technical issues they raise.
4
Electronic Commerce
One of the most popular applications concerns comparative shopping. Some system already provide unique entry points to collections of catalogues. The issue here is the integration of information from the various catalogues. One main difficulty is to “wrap” the various sources to extract their information (nature of the product, price, terms of delivery, etc.) Monitoring is another major issue. Information changes at the sources. In some cases, changes may be so rapid (e.g. prices for air travel) that one prefers to use a “mediator” approach. So, instead of extracting the information and monitoring changes, one asks queries each time a user is looking for information. In many cases still, a warehouse approach with data extracted, stored and monitored is used. Automatic site wrapping and monitoring. The wrapping of data sources is mostly performed today semi-automatically from HTML pages. An engineer examines a collection of pages with similar structure and tailors a program to extract information from it. The weakness of this approach is that it has to remain limited
4
S. Abiteboul
to a reasonably small number of sites and page styles. Also, when the pages layout is modified in a site, the wrapper has to be modified. With XML, the situation is somewhat easier and it is possible to use the information contained in tags to automatically understand the semantics of pages and thus wrap them automatically. When one moves to forms or web services, the situation becomes more intricate since one needs to find which data to input in order to obtain information. Intuitively, an automatic system would have to apply a heavy dose of artificial intelligence to understand the semantics of a web site and automatically extract information from it. Now let us assume that the owner of a site is willing to let others use the information. Then the site owner may “publish” the semantics of the site (e.g., in RDF [14]) and the means to obtain information from it using standards such as WSDL [17] and UDDI [16]. Indeed, one may even expect that entire domains will standardize the way data is published on the web, e.g., NewsML for the news industry. This is raising the issue of the semantic web and is mentioned here primarily to stress that an intelligent monitoring of the web should rely on semantics published by web sites, which seems to take us a bit far into the future.
5
Web Archiving
The content of the web is an increasingly valuable source of information that is worth being archived. Indeed, there are attempts to archive the entire web (Internet web archive [12]) or portions of it. For instance, we are studying with the Biblioth`eque Nationale de France [9] the archiving of the French web [6]. This would be a service to future generations of historians, somewhat like libraries for today’s historians. Even the viewpoint that the web is a huge junk yard suggests the need for archiving it when considering the importance of prehistoric junk yards for today’s archaeologists. Companies may also want to archive their web (private or public) although the motivation is different here: the “memory” of the enterprise is an important asset. Finally, one may want to archive a web site on some specific topics as part of the service provided by the site (e.g., newspaper websites). Better web crawlers. Because of the lack of change control on the web, one issue in constructing web archives is to decide when to version a particular page. The crawlers have to visit the web and detect changes. They must do so by taking advantage of their network bandwidth. Crawling is in principle a trivial process: start from some page in the “core” of the web and follow links to discover new pages. There are subtleties such as avoiding rapid firing, i.e., requesting too rapidly too many pages from the same web server, an action considered unfriendly. There is also the technical challenge of managing billions of URLs. But most importantly the real problems come from the limitations of network-bandwidth compared to the size of the web. Suppose 4 million pages are read per day with a single PC (the rate of Xyleme crawlers for instance); with 25 crawlers, 100 million pages can
Issues in Monitoring Web Data
5
be read per day, which already requires a very large network bandwidth. Still, it takes a month to read 3 billion pages. By that time a lot of the pages that have been read are already stale and reaction to changes is too slow. By accessing once every month the pages of, say the New York Time, a lot of important information is missed. To overcome this problem, one should use the network bandwidth more intelligently. In particular, one should not access all pages with the same frequency. A solution should distinguish between important pages (e.g., pages with high page rank in Google terminology [13] that are read regularly) and less or unimportant pages. A solution should also take into account some estimate of the change frequency of the page (it is not interesting to keep reading an archive) and, last but not least, it should be tailored to the specific goal of the archive. There are many other issues in web archiving. For instance, web archives have motivated a new research on versioning mechanisms [8]. The data we archive is a rapidly changing graph and this raises complex issues. To illustrate, suppose that we take some first-generation crawler to create the archive. Say the crawler uses some breadth-first traversal of the graph of the web. Then a large time interval may occur between the visit of a page and the visit of the pages it points to. These pages are very likely to have changed or may even have disappeared by the time we access them. This is a very important cause of inconsistencies in the archive. Finally, perhaps the most critical technical challenge for such applications is to be able to also archive data from the deep web.
6
Web Surveillance
Web surveillance can be applied in a variety of fields ranging from anti-criminal or anti-terrorist intelligence (e.g., detecting suspicious acquisition of technology) to business intelligence (e.g., discovering potential customers, partners or competitors). Such applications combine web monitoring with data mining and text (or sound or image) understanding. The two main issues from a monitoring viewpoint are (i) to obtain the data (using better web crawlers that were already discussed) and (ii) to filter it (using new query processing technology). Stream and continuous queries. One can distinguish two main approaches: 1. Continuous queries: the same query is evaluated regularly by a query engine and the difference between the answers is analyzed. In other words, we separate data acquisition (crawling and indexing) and query processing; see for instance [5]. 2. Stream queries: streams of data are brought from the web (typically from web robots) and these streams of data are filtered; see, e.g., [7]. Typically, stream queries are more appropriate for fast detection of simple patterns on rapidly changing data (e.g., data that is brought in by sensors) whereas continuous queries are more feasible when considering the detection of complex
6
S. Abiteboul
patterns, e.g., patterns involving joins of data from several sources. In this context, we need to develop new optimization techniques for continuous and stream queries. Also, there is a need for a query language to specify such queries since SQL simply misses the points: It does not provide the necessary constructs for querying streams or specifying continuous queries. The query language should allow to specify a number of aspects such as the pattern that one would like to detect, on what data should the query be performed, how often, what should be done with results (should they be posted on the web, sent by email, how often, etc.) A classical query is in some sense very simple: it is a function that returns a result. A continuous or a stream query is more complex in the sense that it creates in some way a contract between the client and the server, it takes time into account in an essential manner, and also some notion of state of the client (e.g., when we compare the last result of a continuous query to the current one).
7
Mobile Devices
Mobile devices are certainly becoming more and more popular, e.g. laptops, PDA’s, cellular phones, embedded devices, etc. This leads to a vision of lots of data changing rapidly (position of PDA’s) and rather volatile (plugging and unplugging laptops). Perhaps a most critical aspect is geography. For instance, a query such as “where is this particular movie showing” should get a different answer depending on the location of the person asking the query. Indeed, the location and the nature of device used are key components of the user’s profile that should be taken into account in answering queries. These mobile devices should be considered, as participating to the web, e.g., by publishing information on it. The rapidly evolving and massive nature of the web is thus amplified. Furthermore, monitoring is particularly critical in a mobile context, e.g., if I want to be alerted when some cultural events happen nearby or whether a friend happens to be in the neighborhood. To conclude this quick tour of the topic, we consider active web sites.
8
Active Web Sites
In the previous discussion, we somewhat assumed that web sites were “passive” and that some servers were in charge of the surveillance. This is mostly the situation today. However, one should expect the future web sites to include more and more active features (in the style of active databases) that will facilitate friendly web surveillance. In particular, mobile devices should be viewed as peers willing to share data resources with other peers. Bringing active database features to the web. The most important concept in active databases is that of triggers. One would like web sites to be friendly enough to let us install triggers in, say XML documents they manage; see [4]. Suppose I am interested in the price of a particular camera in the Amazone catalogue. I would like to introduce the trigger (specified in an ad-hoc syntax):
Issues in Monitoring Web Data
7
When the price of the Cannon CAN344 changes do email-me More generally, one would like to have web peers that are willing to cooperate using (continuous) methods and offering services such as change notification. Research in that direction is certainly needed; see Active XML [2] for a proposal in this spirit. We are still a long way from truely active web sites. However, web services already provide the technology for enabling more active web sites. To illustrate, consider a bibliography database. It is straightforward to implement a web interface for it using, for instance, SOAP services. A service for notifying the addition of new bibliography references can be easily proposed by combining a database trigger with a call to a SOAP service. Similarly, a service for sending (say once a week) the incremental changes to the database may be easily implemented. What is perhaps more critically missing is a global view of the distributed information system this is leading to. In particular, there are challenging semantics and optimization issues.
9
Conclusion
Let us try to abstract a few main ideas from the applications we considered. The high-level vision we sketched is that the web is moving from a collection of static documents to a world of dynamic knowledge. The knowledge may range from implicit (buried in text or other unstructured media), to somewhat explicit via semantic annotations in the form of XML tags attached to data, to very explicit using future standards such as RDF. One should expect to see more and more semantics specified explicitly on the web, because this is the only feasible path if we are to make effective use of this massive amount of data. In this vision of the web as a world-wide knowledge base, an essential aspect is that the knowledge changes in time. Web knowledge has to be monitored to make available a timely version of information. Acknowledgment We would like to thank all the people involved in the Xyleme project [11] and the Xyleme company (www.xyleme.com). We would also like to thank those who work on Active XML and in particular, Omar Benjelloun, Angela Bonifati, Ioana Manolescu, Tova Milo, and those who discussed with me the French web archive, Gregory Cobena and Julien Massanes. Finally, we thank Victor Vianu for his comments on a first draft of this paper.
8
S. Abiteboul
References 1. S. Abiteboul, P. Buneman, and D. Suciu. Data on the Web. Morgan Kauffman, 1999. 2. Serge Abiteboul, Omar Benjelloun, Tova Milo. Towards a Flexible Model for Data and Web Services Integration, proc. Internat. Workshop on Foundations of Models and Languages for Data and Objects, 2001. 3. Brian Babcock; Shivnath Babu; Mayur Datar; Rajeev Motwani; Jennifer Widom. Models and Issues in Data Stream Systems, PODS, 2002. 4. Angela Bonifati, Daniele Braga, Alessandro Campi, Stefano Ceri: Active XQuery. ICDE 2002 5. Jianjun Chen, David J. DeWitt, Feng Tian, Yuan Wang: NiagaraCQ: A Scalable Continuous Query System for Internet Databases. SIGMOD Conference 2000: 379390 6. S. Abiteboul, G. Cob´ena, J. Masanes, G. Sedrati, A First Experience in Archiving the French Web, to appear in ECDL, 2002. 7. Benjamin Nguyen, Serge Abiteboul, Gr´egory Cobena, Mihai Preda, Monitoring XML data on the Web, SIGMOD, 2001. 8. Amelie Marian, Serge Abiteboul, Gr´egory Cobena, Laurent Mignet, ChangeCentric Management of Versions in an XML Warehouse, VLDB, 2001. 9. Julien Masan`es, The BnF project for Web archiving, Preserving online content for future generation, ECDL Workshop, 2001. 10. Raghavan, S. and H. Garcia-Molina. Crawling the Hidden Web. in 27th International Conference on Very Large Data Bases. 2001. Rome, Italy. 11. Lucie Xyleme (a nickname for the people who worked on the Xyleme project). A dynamic warehouse for XML Data of the Web. IEEE Data Engineering Bulletin 24(2): 40-47 (2001) 12. The Internet Archive, www.archive.org 13. Google, www.google.com 14. Resource Description Framework, www.w3.org/RDF 15. Simple Object Access Protocol (SOAP) 1.1, http://www.w3.org/TR/SOAP/ 16. Universal Description, Discovery and Integration of Businesses for the Web, www.uddi.org 17. Web Service Definition Language, www.w3.org/TR/wsdl 18. XQuery 1.0: An XML Query Language, http://www.w3.org/TR/xquery 19. Extensible Markup Language (XML), www.w3.org/XML 20. Xyleme SA, www.xyleme.com 21. The World Wide Web Consortium, www.w3.org
Controlled Caching of Dynamic WWW Pages Costas Vassilakis and Giorgos Lepouras University of Athens, Department of Informatics and Telecommunications Panepistimiopolis, TYPA Buildings, Athens 157 71 Greece {C.Vassilakis, G.Lepouras}@di.uoa.gr
Abstract. In order to increase flexibility and provide up-to-date information, more and more web sites use dynamic content. This practice, however, increases server load dramatically, since each request results to code execution, which may involve processing and/or access to information repositories. In this paper we present a scheme for maintaining a server-side cache of dynamically generated pages, allowing for cache consistency maintenance, without placing heavy burdens on application programmers. We also present insights to architecture scalability and some results obtained from conducted experiments.
Introduction Content offered by WWW servers can be classified in two categories, static and dynamic. Static content comprises of HTML pages and images stored in files; dynamic content is created by the WWW servers upon receipt of a relevant request, through the execution of server-side program/script (CGI, ASP, JSP etc). Usually, these programs accept some parameters, included in the request and extract data from an information repository (e.g. a database) to formulate the HTML page that will be sent back to the requestor. While in the early days of WWW dynamical content was rare, its share is constantly increasing due to the need for increased flexibility in HTML page formulation and the desire to provide up-to-date information. Dynamic content, however, increases server workload, since it necessitates the execution of a (possibly costly) code fragment, while additional overheads (environment setup, housekeeping etc.) are inherent in such an approach. This workload may prove overwhelming for heavily accessed servers, and techniques used for static HTML pages (e.g. client-side caching, proxy servers) may not be employed for reducing it. This is due to the fact that such techniques do not take into account the particularities of dynamic content, such as the existence of parameters and the possibility that either data, or even the program that were used to formulate the page may have changed in the meantime. This paper presents a scheme for maintaining a server-side cache for dynamic content, which may reduce drastically server workload, without compromising the advantages offered by the usage of dynamic content. In the rest of this paper, section 2 covers related work and outlines open issues. System architecture is presented in section 3, and cache consistency maintenance is discussed in section 4. Section 5 focuses on scalability and resource utilization and section 6 presents experimental results. Section 7 concludes and outlines future work.
R. Cicchetti et al. (Eds.): DEXA 2002, LNCS 2453, pp. 9–18, 2002. Springer-Verlag Berlin Heidelberg 2002
10
C. Vassilakis and G. Lepouras
Related Work In [11] a cost model for the materialization of web views, i.e. subsets of a server’s dynamically computed information content, is presented. In [12] pre-generation of Web pages is discussed and changes in the information repository are addressed, but no algorithm for selecting the most appropriate pages for caching is presented. Caching of dynamic content is also discussed in [3], [13] and [6] and [9] presents techniques for reducing a web server’s workload, regarding dynamic content creation. In [15], two categories of dynamic content caching are described, active cache and server accelerators. In active cache [4] servers supply cache applets that are attached to documents and are executed when a user “hits” a cached object. Although this scheme reduces network load, it has a significant CPU cost. On the other hand, web server accelerators reside in front of web servers. Such techniques were used to improve performance at the Web Site for the 1996 Olympics [10] and for the 1998 Winter Games [6]. In the 1998 Games the Data Update Algorithm was used to maintain data dependence information between cached objects and the underlying data, enabling cache hit rates close to 100% compared to 80% of an earlier version. However, this approach suffers when it comes to web sites that offer a large number of dynamic pages [19], because it depends on keeping up to date a fine-grain graph that describes dependencies among each web page and the underlying data. In [19] a different approach is proposed, where the caching mechanism resides behind the web server and the fine grain method described in [6, 10] is supplemented with coarse grain dependencies between data and groups of dynamic pages. The caching algorithm uses a URL class based invalidation method and selective precomputing. Commercial products implementing caching of dynamic content exist as well. Cold Fusion [1] offers the capability to designate programs that produce non-changing dynamic pages, and Cold Fusion engine arranges so that outputs from this program are cached and reused, while XCache [18] can be integrated into the IIS Web server to maintain a cache of dynamically generated Pages, providing an additional API for update applications so as to inform the cache engine for outdated pages. All techniques presented insofar, however, require from application programmers an amount of additional development and maintenance for adapting programs to the caching scheme, for reasons of update propagations and consistency management. Moreover, invalidated pages are designated using proprietary specifications or APIs, increasing thus overall system complexity. Finally, most implementations are coupled to specific web servers, reducing interoperability and portability.
Maintaining a Cache of Dynamically Generated Pages A server-side cache of dynamic content may be maintained in a controlled fashion using the architecture depicted in Figure 1. The proposed scheme complements the existing web site installation with two additional software modules, the cache manager and the update manager, which are responsible for maintaining the cache and ensuring that the cache is consistently updated or invalidated, when data is modified. The proposed scheme places the new modules as a front end to the existing installation, without any need to affect the latter; some additions may only be needed
Controlled Caching of Dynamic WWW Pages
11
for update programs, as explained later. This characteristic was one of the design goals, since site administrators would be reluctant to employ a scheme that would require major changes to their site.
Fig. 1. Architecture for maintaining a cache of dynamic content
Upon start-up, the Cache Manager reads its configuration, which is derived from the class-based cache management for dynamic web content, described in [23]. Under this scheme, web pages are grouped into classes, based on page URLs and client information. The URL information used for classifying pages includes network paths, program names and parameters. Client information may complement the specification of a class and include cookies, client domain names or IPs, browser information (e.g. HTTP_ACCEPT_ENCODING, HTTP_USER_AGENT), or any other piece of information designated in the HTTP protocol for the server-side program environment. Figure 2 presents example cache specifications for three classes. We note that our approach does not use the Dependence specification described in [23], since a different mechanism (see section 4) is used for change tracking and page invalidations. URL-Class: http://www.example.org/php/comments.php?op=submit Cachable: No Server: local-www-srv:80 URL-Class: http://www.example.org/php/search.php Cachable: no Server: local-www-srv:80 URL-Class: http://www.example.org/php Cachable: yes Page-ID:_cookie:LANGUAGE Server: local-www-srv:80 Fig. 2. Cache specification for two document classes
The first class designates that pages generated by server-side program (SSP) comments.php with the parameter op set to the value submit are not eligible for caching (due to side effects of execution); the second class denotes that search results
12
C. Vassilakis and G. Lepouras
are not cacheable either (due to the small probability that two users will use the same search text). The third class defines that pages generated by SSPs within the php catalogue are cacheable, but additionally states that the page contents depend on the value of the LANGUAGE cookie. In all three cases, the Server lines (an addition to the specifications presented in [23]) designate that the web server producing these pages is named local-www-srv and may be contacted at port 80. The name local-www-srv need not be known to the clients, since only the Cache Manager uses it. After initialisation, the Cache Manager awaits for web requests. For each request, it checks whether the requested page has been cached or not. A page, in the context of dynamic content, is identified by the URL, the parameters passed to the SSP and any cookies and HTTP variables pertaining to the class in which the request falls. If the page is found in the cache, it is returned to the client immediately. If not, the Cache Manager requests the page from the web server and forwards the results to the client. In order to rapidly determine if the target page exists in the cache, the Cache Manager employs the algorithm described in [16], which computes an MD5 checksum of the requested URL and matches it against the checksums of the cached pages. Subsequently, the Cache Manager inspects its configuration to determine whether the page just served is eligible for caching. In this case, the page is placed in the cache, so that it becomes available for later requests. If the page is not eligible for caching, the results returned to the client are simply dropped. In the case that the cache is full and a page needs to be inserted to it, the Cache Manager removes some previously cached pages from the cache to make space available for the new page. The page(s) that will be removed are determined using the Greedy Dual-Size [5] algorithm, and more specifically the variation aiming to maximize the hit ratio, since it has been found to achieve superior performance, compared to other algorithms within the context of web caching [2]. The Cache Manager cooperates with the Update Manager module, described in section 4, to guarantee that the pages sent to the clients are not outdated, due to changes in the data or the server-side programs that generate the pages.
Handling Data and Program Changes Pages generated by SSPs and stored within the cache may become obsolete, since the underlying data, or even the server-side programs themselves may be modified. In these cases, the affected pages in the cache must be located and either be removed, or re-generated. The responsibility for update tracking is assigned to the Update Manager, which is a distinct software module that may be run on the same machine as the cache manager or on a different one. The Update Manager monitors the invalidations repository, to locate patterns describing the pages that have become obsolete. Each such pattern contains the following information: 1. A designation of the program(s) that have generated the affected pages. 2. The parameters and the respective values that must match the server-side program’s invocation data, so as to consider the page invalid. Figure 3 presents some examples of invalidation patterns. Each invalidation pattern is complemented with a criticality designation, which draws values from the domain {hard, soft}. A value of soft indicates that the outdated version of the page may be used as a reply to the requesting client, in the event of high
Controlled Caching of Dynamic WWW Pages
13
server load, whereas a hard criticality designation states that sending outdated pages is unacceptable, and the respective cached pages should be invalidated immediately. Updates having a soft criticality designation are assigned to a low-priority thread of the Cache Manager, which uses idle CPU cycles to refresh outdated pages. (Path, Parameters) (“http://www.example.org/categories.php”, “catid=2”)
(“http://www.example.org/admin/*”, “”) (“http://www.example.org/index.php”, “__cookie:LANGUAGE=en”)
Remarks All pages generated by categories.php with the parameter catid equal to 2, regardless of the values of other parameters or cookies. All pages generated by any program in the admin directory, regardless of parameters. All pages generated by index.php with the cookie LANGUAGE set to the value en.
Fig. 3. Examples of patterns within the invalidations repository
Once the Update Manager detects a new invalidation entry within the invalidations repository, it reads the respective data and propagates them to the Cache Manager, so that the respective page invalidations or regenerations may be performed. The Cache Manager communicates with the Update Manager using a private protocol, which is not needed to be known to the applications that modify the data repository, contrary to the practice employed in [23] and [10]. These applications need only insert the corresponding invalidation entries to the invalidations repository (which may be a database table, an operating system file etc.), using familiar programming techniques. Using an additional repository for registration of invalidation patterns and the introduction of the update manager were design decisions, aiming to relieve the update application programmers from using proprietary APIs, the need to know details about the number of Cache Managers and their addresses, or handling communication errors. Furthermore, using a standard information repository for invalidations, such as a database, facilitates interventions from the Web administrators that could not be foreseen at the time of update application development. For instance, in order to force the invalidation of all pages generated by the /categories.php script (e.g. due to changes in the script), the Web administrator may append the respective entry to the invalidations repository by using e.g. SQL: insert into cache_invalidations(script_path, params, criticality) values('http://www.example.org/categories.php', '', 'hard');
If invalidation information were communicated to the cache manager through an API, it would be necessary to write compile and execute a –simple– program that would arrange for sending the respective information. Moreover, invalidation records use a format very familiar to web programmers and administrators; therefore it is expected that web site maintenance will run more smoothly, compared to cases that employ proprietary protocols or specifications. Finally, if the data repository supports active features, application programmers may be totally relieved from the burden of registering the invalidation patterns within the invalidations repository. This approach is outlined in section 4.2.
14
C. Vassilakis and G. Lepouras
Locating Affected Pages within the Cache When the Cache Manager receives an invalidation notification, it must locate all pages matching the patterns for the script path and the parameters and either remove them from the cache or refresh them (in immediate or deferred mode, as specified by the criticality designation). Since the model presented in this paper allows for specifying parameter subsets or wildcards within the script paths, the MD5 checksum of the URL, stored for the purpose of serving client requests, cannot be used to locate the affected pages, since hash functions can only be used for full key search [8]. To avoid scanning the full table of cached pages, which would be expensive, the following techniques are used, together with the primary MD5 checksum hashing: 1. An additional MD5 checksum is computed on the program path portion of the URL for each page within the cache, and a hash structure is maintained based on this checksum. This hash structure is used to locate the pages generated by a specific server-side program, when the script path field of an invalidation entry does not contain wildcards. For each of these pages, standard substring searching is used to further filter pages matching the specified parameters, if necessary. 2. A secondary B-tree index is built on the script path field of the cached pages. This index is used to locate the affected pages when the invalidation entry’s script path field does contain wildcards. Since the maintenance of this index is more costly than the maintenance of MD5 hash tables, an administrative option is provided to disable it; in this case, however, wildcard matching in script paths is not supported. 3. An optimisation hint may be attached to an invalidation entry, specifying that the page designation is complete, i.e. it includes all parameters to the SSP. In this case, the primary MD5-checksum hashing algorithm may be used to locate the single page that has become obsolete, minimizing thus housekeeping overheads. Separating Invalidations from Update Applications In the approaches presented in the literature insofar, application programs that modify the data repository need to notify in some way the Cache Managers regarding the pages that have become obsolete. This is a burden for programmers and introduces another source of application maintenance activities, when the mapping between data items and dependent web pages changes. In these cases, all programs modifying the respective data need to be tracked and updated. The approach presented in this paper allows for exploitation of active features offered by the data repository to relieve programmers from the extra coding and maintenance. Such active features may be found in many DBMSs, e.g. triggers in Oracle, rules in INGRES etc. These active features may be used to implement automatic entry addition to the invalidation repository when the underlying data changes. For instance, the PL/SQL code create trigger category_invalidations after insert or delete on stories for each row begin insert into cache_invalidations (script_path, params, criticality) values('http://www.example.org/categories.php', 'catid=' || catid, 'hard'); end;
Controlled Caching of Dynamic WWW Pages
15
arranges for automatically inserting into the cache invalidations repository the appropriate record for invalidating the pages generated by the script categories.php, when its catid (category id) parameter matches the category id of an inserted or deleted item in the stories table. Similar results may be obtained by using stored procedures within the information repository, in order to modify the data, instead of direct SQL (or other information repository-dependent) statements. For instance, the addition of a story may be performed using the statement exec procedure insert_story(800, 3, 'New story', 'Very interesting');
where insert_story may be defined as create procedure insert_story(sid number, cid number, title varchar2, body varchar2) as begin insert into stories values(sid, cid, title, body); insert into cache_invalidations(script_path, params, criticality) values(' http://www.example.org/categories.php', 'catid=' || cid, 'hard'); end;
Both the trigger-based and the stored procedure-based techniques are feasible because invalidations are stored within a data repository, instead of being communicated to the Cache Manager via a custom interface, which –in generalcannot be directly used from within a database. Moreover, both cases present the additional advantage of centralising the mapping between updated data and affected pages; thus when this mapping changes, only a single modification is required (within the database schema), rather than one update for each affected application.
Architecture Scalability and Resource Utilisation When the number of requests to the web site increases, the cache manager may become the performance bottleneck of the installation. This may be tackled by using an array of cache managers rather than a single one, as depicted in Figure 4. Client requests may be distributed amongst the servers within the array using any appropriate technique (e.g. clustering, round-robin DNS, or specialised hardware [7]). In this configuration, cache managers maintain independent caches, serving client requests using the algorithm described in section 3. Although in some cases it would be preferable for a Cache Manager to fetch a pre-generated page from a sibling cache, rather than from the web server, the current implementation does not support ICP [19], cache digests [14], or any protocol for communication between cache servers. When updates to underlying data take place, the invalidations repository is populated with the appropriate entries, as presented in section 4. The Update Manager monitors new additions and arranges for communicating the relevant data to the Cache Managers, whose addresses are registered within the Update Manager’s configuration file. Each Cache Manager proceeds then to the invalidation of the affected pages, following the algorithm described in section 4.1. The Update Manager is the sole responsible for handling communication errors, retransmissions to failed nodes etc.
16
C. Vassilakis and G. Lepouras
Cache Manager 1
Cache Manager 2
...
Cache Manager n
Invalidation notifications Update manager
Invalidations repository
Data repository/ update applications
Fig. 4. Load balancing through introduction of multiple caching servers
We would like to note that although the web server, the Cache Manager and the Update Manager appear as distinct modules in Figure 4, it is possible that two of these modules (or even all of them) run on the same machine, in order to save resources. In medium configurations it is expected that only one Cache Manager will be used and the Update Manager will be hosted in the same machine as the cache manager. In large installations, the Cache Manager might be placed in a separate machine. Finally, a single Cache Manager may be used to cache multiple web sites, if this is desirable. Experiment Results In order to assess the various aspects of our approach, a number of experiments were conducted. The experiments were selected so as to vary in different aspects, such as update frequency, average page size, update propagation criticality and the existence of personalised pages. For each experiment, stress tests were performed under different loads, so as to quantify the benefits derived from using the caching scheme. The first experiment was conducted on a web portal implemented using the PHPNuke system. In this experiment, a number of stories were added daily; when a story was added, the main page (/index.php) should be refreshed instantly, whereas pages hosting news falling in specific categories (/categories.php) were updated with softer time constraints. Personalised pages (implemented via cookies) were not cached, as log analysis showed that each such page was used only 1.2 times in average. The second experiment was conducted for a sports site, offering “live” information about the scores of soccer and basketball games. Pages served from this site were classified in three categories: (a) a summary page for all soccer games (b) a summary page for all basketball games (c) one detail page for each match (basketball or soccer). In this experiment, the soccer summary page was updated once every two minutes, in average, whereas the basketball summary page was updated constantly as game scores evolved. Out of these two summary pages, only the first one was cached. Regarding detail pages, the average update rate for soccer pages was once every 3’:30”, while the corresponding rate for basketball match pages was once every 28”. Refresh information sent to clients dictated page refreshing every 30” for both soccer and basketball pages. “Impatient” users, however, often requested page refreshes every 3”-5”, especially during the last few minutes of games.
Controlled Caching of Dynamic WWW Pages
17
Both experiments were conducted using two site configurations. In the first configuration, a single machine hosted all software modules, i.e. the web server, the Cache Manager and the Update Manager. In the second configuration, two machines were used, with the first one running the Cache Manager, and the second one hosting the Update Manager and the web server. The results from these experiments are summarised in Figure 5. Single-server # clients Web portal Sports site Dual-server # clients Web portal Sports site
Cache hit ratio 250 500 73% 78% 60% 65% Cache hit ratio 50 250 500 62% 73% 78% 43% 60% 65% 50 62% 43%
Avg. response time reduction 50 250 500 61% 72% 67% 51% 62% 59% Avg. response time reduction 50 250 500 59% 70% 76% 49% 59% 66%
Fig. 5. Experiment results using a dual-machine configuration
In these results, it is worth noting that in the single-machine configuration, although the cache hit rate increases when the number of users grows from 250 to 500, the average response time reduction drops. This is due to server saturation, since the server cannot handle all three tasks under this heavy load. The situation is remedied when the cache manager is separated from the web server, in the second configuration, where the load is balanced between the two servers.
Conclusions – Future Work This paper proposes a caching scheme for dynamic web pages, which isolates the caching mechanism from the web server. Although this separation duplicates the parsing of URLs and headers, it presents a number of advantages: • the caching server can interoperate with any web server, thus no web-server specific implementations are required • the two servers (web and caching) may reside on different systems • any upgrade/modification to any of the servers does not affect the others. The architecture presented minimises the additional workload for dynamic page developers, by allowing separation of the rules describing the dependencies between underlying data and derived pages from the application programs that perform the data updates. These dependencies can be stored within the database schema, in a centralised fashion, leaving update applications intact. This requirement can be easily met when the database offers triggering mechanisms, or stored procedures. The caching scheme was tested in several cases, proving quite efficient and well behaved in cases of information repository or server-side program updates. The results showed that the proposed caching mechanism offers improved network performance with minimum CPU overhead. It should, however, be noted that the results are dependent on the nature of the dynamic pages. If the underlying data change too often (e.g. stock exchange), the caching scheme will probably not improve
18
C. Vassilakis and G. Lepouras
the web site performance. Further research is underway to form a set of guidelines and tools to aid developers in assessing the suitability of dynamic content for caching. Future work will focus on supporting cache population using anticipatory rules, to cater for expected access patterns over time periods, and extending cache techniques to the client side, so as to limit the required network traffic. Cooperation between sibling caches, and its effect on performance is another area that will be investigated.
References 1. Allaire Corporation: Cold Fusion White Paper. Version 4.0. (1999). http://www.allaire.com 2. Arlitt M., Friedrich R., Jin T.: Performance Evaluation of Web Proxy Cache Replacement Policies. Proceedings of the 10th International Conference, Tools ’98, Palma de Mallorca, Spain, (1998) 193-206 3. Atzeni P., Mecca G., Merialdo P.: Design and Maintenance of Data Intensive Web Sites. Proceedings of the Conference of Extending Database Technology, Valencia, Spain (1998) 4. Cao P., Zhang J., Beach K. Active Cache: Caching Dynamic Contents on the Web. Proceedings of IFIP International Conference on Distributed Systems Platforms and Open Distributed Processing (Middleware '98). (1998) 373-388. 5. Cao P., Irani S.: Cost-Aware WWW Proxy Caching Algorithms. Proceedings of USENIX Symposium on Internet Technologies and Systems. Monterey, California (1997) 193-206 6. Challenger J., Iyengar A., Dantzig P. A Scalable System for Consistently Caching Dynamic Web Data. Proceedings of the IEEE Infocom 99 Conference. New York, (1999) 7. Coyote Point Systems Inc. The Equalizer Network Appliance. Available through http://www.coyotepoint.com/equalizer.shtml 8. Elmarsi R., Navathe S. Fundamentals of Database Systems. Benjamin/Cummings Publishing Company Inc. (1994) 9. D. Florescu et al. Run-Time Management of Data Intensive Web-sites. Proceedings of the Workshop of Web and Databases (WebDB 99), Philadelphia, Pensylvania. (1999) 10.Iyengar A., Challenger J. Improving Web Server Performance by Caching Dynamic Data. Proceedings of the USENIX Symposium on Internet Technologies and Systems, Monterey, California (1997). 11.Labrinidis A., Roussopoulos N. On the Materialization of Web Views. Proceedings of the Workshop of Web and Databases (WebDB 99), Philadelphia, Pensylvania (1999) 12.Pröll B. et al. Ready for Prime-Time – Pre-Generation of WEB Pages in TIScover. Proceedings of the Workshop of Web and Databases, Philadelphia, Pensylvania (1999) 13.Sindoni G. Incremental Maintenance of Hypertext Views. Proceedings of the Workshop of Web and Databases (1998) 14.Hamilton M., Rousskov A., Wessels D. Cache Digest specification - version 5. Available through http://www.squid-cache.org/CacheDigest/cache-digest-v5.txt 15.Wang J. A Survey of Web Caching Schemes for the Internet. ACM Computer Communication Review, (29) (1999) 36--46 16.Wessels D. Squid Internet Object Cache. http://www.squid-cache.org 17.Wessels D., Claffy K. Internet Cache Protocol (ICP), version 2 – RFC 2186. Available through http://www.cis.ohio-state.edu/cgi-bin/rfc/rfc2186.html 18.XCache Technologies, XCache Product Information. http://www.xbuilder.net/home/ 19.Zhu H., Yang T. Class-based Cache Management for Dynamic Web Content. IEEE INFOCOM, 2001.
Deriving and Verifying Statistical Distribution of a Hyperlink-Based Web Page Quality Metric Devanshu Dhyani, Sourav S. Bhowmick, and Wee Keong Ng School of Computer Engineering, Nanyang Technological University, Singapore 639798 {assourav, awkng}@ntu.edu.sg
Abstract. In this paper, we discuss issues related to metrics for Web page quality. These metrics are used for ranking the quality and relevance of Web pages in response to user needs. We focus on the problem of ascertaining the statistical distribution of some well-known hyperlink based Webpage quality metrics. Based on empirical distributions of Webpage degrees, we derived analytically the probability distribution for the Pagerank metric. We found out that it follows the familiar inverse polynomial law reported for Webpage degrees. We verified the theoretical exercise with experimental results that suggest a highly concentrated distribution of the metric.
1
Introduction
The significance of a Web page can be viewed from two perspectives—its relevance to a specific information need such as a user query, and its absolute quality irrespective of particular user requirements. Relevance metrics relate to the similarity of Web pages with driving queries using a variety of models for performing the comparison. Quality metrics typically use link information to distinguish frequently referred pages from less visible ones. Specifically, a well-known set of metrics for measuring the quality of Web pages is based on hyperlink citations, for example Pagerank [2] and Authorities/Hubs [4]. As these measures depend upon Web page in and out-degrees, knowledge of degree distribution can lead to their probability density functions. In this paper, we study the measurement of hyperlink information at a microscopic level in assessing the quality or relevance of page. We demonstrate an approach for deriving the distribution of Pagerank from the empirical distributions of topological primitives. We also experimentally verify the probability distribution of Pagerank. There are several reasons why this exercise is instructive. Firstly, it illustrates a generic methodology that can be extended to other hyperlink metrics. Secondly, a distribution derived theoretically from observations of more primitive determinants is likely to be more reliable than an empirically obtained one that is inextricably linked to the experimental setup. This conforms with the conventional wisdom of making measurements as fundamental as possible before deriving more comprehensive metrics. Additionally, in the case of Web hyperlink metrics such as Pagerank, we R. Cicchetti et al. (Eds.): DEXA 2002, LNCS 2453, pp. 19–28, 2002. c Springer-Verlag Berlin Heidelberg 2002
20
D. Dhyani, S.S. Bhowmick, and W.K. Ng
avoid running computationally expensive algorithms. Finally, a theoretical distribution serves as a model that can help us predict precisely and consistently the effect of changes in certain parameters without incurring the cost of carrying out complex measurements again.
2
Distribution of Quality Metrics
Quality measures depend upon Web page in and out-degrees, knowledge of degree distribution can lead to their probability density functions. Knowing, say the cumulative distribution of Pagerank for the Web FR , one can determine the number of high-quality pages according to some threshold say r, given the size of the Web N . That is, the number of pages with Pagerank greater than r can be estimated as N = N · Pr(R > r) = N (1 − FR (r)). The value of N can be useful for crawlers looking for high quality Web pages in deciding optimum size versus quality configurations for search engine indexes. The distribution can also help Web crawlers give greater priority to visiting more important, high quality pages first. In this section, we demonstrate an approach for deriving the distribution of Pagerank from the empirical distributions of topological primitives. We first revisit the formulation of Pagerank and explain its key determinants. The Pagerank Ri of a page i having in-degree n can be defined in terms of the Pagerank of each of the n neighboring pages and their out-degrees. Let us denote by j (1 j n), the index of neighboring pages that point to i and by Xj the outdegree of page j. Then for a fixed parameter d in [0,1] the Pagerank Ri of i is n R given as Ri = (1 − d) + d j=1 Xjj . We refer to d (0 d 1) as the damping factor for the calculation of Pagerank. We derive the distribution of a simplified version of Pagerank, ignoring the recurrent relationship of Ri with the Pagerank of other pages Rj and assuming the formulation to be Ri = (1 − d) + d
Ni 1 Xj j=1
(1)
Computationally, the determination of Pagerank for a graph of k pages can be seen as equivalent to the steady state solution (n → ∞) of following matrix product relationship: 1 1 1 x11 · · · xi1 · · · xk1 n n+1 1 1 1 R1 1−d R1 · · · · · · x x x i2 k2 n 1 − d 12 R2n+1 R 2 .. = .. + d .. .. .. · .. . . . . . . 1−d Rkn Rkn+1 1 1 1 · · · · · · x1k xik xkk where Rin denotes the Pagerank of page i at the nth iteration and xij the number of links from page i to j. If there are no outgoing links from page i to j,
Statistical Distribution of a Hyperlink-Based Web Page Quality Metric
21
i.e., xij = 0, then the corresponding entry in the matrix (1/xij ) is set to zero. Repeated multiplication of inverse out-degree matrix with the Pagerank vector yields the dominant eigenvector of the latter. Pagerank can thus be seen as the stationary probability distribution over pages induced by a random walk on the Web, that is, it represents the proportion of time a “random surfer” can be expected to spend visiting a page. It is clear that the steady state distribution of the Pagerank vector (Rin ) depends entirely on the value of d and the right hand vector of inverse out-degrees. Our simplification of Equation 1 aims at finding the distribution of this vector at n = 0. The distribution of (Ri1 ) gives us an idea of the steady state distribution at n → ∞ which can itself be obtained by applying the above computation on the initial distribution. We further assume that initially Pagerank is uniformally distributed, that is, Ri0 = 1/k for all i (1 i k). We interpret Ri , Xj and Ni in Equation 1 as random variables denoting Pagerank of i, the out-degree of j and the in degree of i respectively. Although both Xj and Ni are known to have the same distribution, Xj is continuous while Ni is discrete. It is clear that Ri for all values of i are identically distributed. The same holds for the in- and out-degrees denoted by Xj and Ni . We therefore represent the common probability densities of these sets of random variables as fR (r), fX (x) and fN (n) respectively. The problem now is to find the density fR (r) given the relationship of Ri with Xj and Ni as represented by Equation 1. 2.1
The Lotka Density
The derivation of the distribution of Pagerank is based on observations of distribution of Web page degrees. These measurements carried out on Webgraphs by Broder et al [1] and Kleinberg et al [5] have been reported to follow the well-known Lotka distribution. The Lotka density is given as C if x 1, α x fX (x) = 0 otherwise. where α ≈ 2 and C is a constant. In our derivation, we invoke both the continuous and discrete versions of this law. Here we distinguish between the two and examine the implications of each. If we interpret X as a continuous random variable, the constant C is found using the
∞ fact that the area under a probability density curve sums to unity. That is, −∞ fX (x)dx = 1. Applying this to the continuous Lotka density above, we have ∞ C dx = 1 (2) 2 x 1 Solving this we obtain C = 1. The continuous version of Lotka’s law can then simply be stated as follows: 1 if x 1, 2 fX (x) = x 0 otherwise.
22
D. Dhyani, S.S. Bhowmick, and W.K. Ng
In the case of the discrete version, the integral of Equation 2 changes to ∞ the constant C from a discrete summation, hence, x=1 xC2 = 1. If we factor ∞ −2 the summation on the left, we are left with the sum which is the x=1 x well-known Riemann zeta function ζ(2). Several analytical methods exist for computing the zeta function. Here we use the following general definition for even n−1 n n |π where |Bn | is a Bernoulli number. arguments, i.e., n ≡ 2k, ζ(n) = 2 |B n! π2 Given B2 = 1/6, we have for n = 2, ζ(2) = 6 . Substituting this we obtain the 1 expression for C as C = ζ(2) = π62 ≈ 0.608. Although the in- and out-degree distributions on the WWW have been discovered principally as discrete distributions, we apply the continuous approach in examining the relationship between degree and Pagerank distributions because the apparatus of infinitesimal calculus makes the mathematical formulation easier. Indeed, as the number of pages being considered increases, the differences between the continuous and discrete approaches become insignificant. To approximate the Lotka distribution function FX (x) = Pr(X x), we integrate the density
x 1 function 1fX (·) within the continuous range (1, ∞): FX (x) =
x f (z)dz = dz = 1 − x . 1 X 1 z2 2.2
The Pagerank Distribution
The non-recurrent definition of Pagerank in Equation 1 may be viewed as a composition of three primitive functions of random variables enumerated below. The inverse of individual out-degrees of pages: The individual out-degrees are independent identically distributed random variables, Xj with the index j ranging from 1 to the in-degree of the page being considered. If we represent the inverse of the out-degree of the j th neighboring page as a random variable Yj then, Yj = X1j . It is known that the density of out-degrees, fX (x) is the Lotka function introduced earlier. We denote the density function of Yj as fY (y), since Yj is identically distributed for all j. The sum of out-degree inverses: We denote Ni by a random variable Zi , the sum Yj . The upper limit to the sum is of out-degree inverses Yj . That is, Zi = j=1 itself a random variable denoting the out-degree of the page in question. Fortunately, this random variable Ni , the out-degree of page i, is Lotka distributed. We must note however, that Ni is necessarily a discrete random variable as must any index to a discrete summation. Thus, Ni has the probability density obtained earlier for the Lotka distribution 6 1 if n 1, 2 n2 π fN (n) = 0 otherwise. Finally, the Pagerank function of Equation 1 can be expressed as a linear function of the random sum Zi above as Ri = (1 − d) + dZi
(3)
Statistical Distribution of a Hyperlink-Based Web Page Quality Metric
23
We now determine the densities of the random variables Yi , Zi and Ri introduced above. Consider Yi , the inverse of Xi that represents the out-degree of page i. We first note that Yi is a strictly decreasing function in the range of Xi (with the latter defined on positive values only). The probability distribution of Yj can be expressed in terms of the distribution of Xj as follows FY (y) = since X1j is strictly decreasing = Pr(Yj y) = Pr X1j y = Pr Xj y1 1 − Pr Xj < y1 = 1 − FX y1 . Differentiating this to convert to probability d d 1 1 1 densities, we have: fY (y) = dy = FY (y) = −fX y1 dy f y y 2 X y . Substituting the Lotka continuous density function for fX (·) to the above result and applying the range [1, ∞] to the argument 1/y, we obtain the uniform density for Yj over the converted range (0, 1]. 1 if 0 < y 1, fY (y) = 0 otherwise.
The above result, that the inverse of a Lotka distributed random variable has a uniform distribution is an interesting coincident. Intuitively it implies that even though the probability that the out-degree of a page is in a given range follows an inverse square law, the probability of out-degree inverse is uniformally distributed, i.e., independent of the value of Yj . n The sum of out-degree inverses Zi given by j=1 Yj has a variable number of terms equal to n, the number of pages that point to i or the out-degree of i. We model the limit of the summation itself as a discrete random variable Ni . Such a sum is commonly referred to as a random sum. As noted earlier, Ni is Lotka distributed and similar to Xi except that the distribution here is the discrete version of the Lotka function. Note that Zi for all values of i are identically distributed so the common density can be denoted fZ (z) as done earlier for the random variables Xi and Yi . We first find the density of Zi conditioned on the summation limit Ni , that is fZ|N (z|n). A sum of random variables has a density which is the convolution of the densities of individual random variables. For n identically distributed summands, this specializes to the n-fold convolution of their common density, in this case (n) (1) fY (y). This n-fold convolution fY (y) is defined recursively as follows: fY (y) =
∞ (n) (n−1) fY (y) and fY (y) = −∞ fY (y − u)fY (u)du for n > 1. The above definition can be applied to fY (y) derived earlier to obtain the following formula for the density of the random sum Zi conditioned upon the out-degree of page i. Figure 1 shows the n-fold convolutions of the uniform density for several values of n. Observe that for higher values of n, the curve flattens out resembling a normal distribution. This is predicted by the central limit theorem which states that the sum of n independent random variables tends to a uniform distribution as (n) n → ∞, fZ|N (z|n) = fY (y) where
24
D. Dhyani, S.S. Bhowmick, and W.K. Ng
Convolutions 1 n=2
0.8 n=4
0.6
n=6 n=8
0.4
n = 10
0.2 0
0
1
2
3
4
5
6
7
8
9
10
Fig. 1. Convolutions of n uniform densities.
(n)
fY (y) =
x 1 j n (x − j)n−1 if 0 < x < n, (−1) j (n − 1)! j=0 0
otherwise.
By the law of total probability, the continuous marginal density of the sum ∞ (n) Zi can be found as fZ (z) = n=1 fZ|N (z|n) · fN (n). Substituting for the above expression for fZ|N (z|n) and the discrete Lotka density for fN (n) we have x ∞ 1 j n 6 (x − j)n−1 if 0 < x < n, (−1) j fZ (z) = π 2 n=1 n2 (n − 1)! j=0 0 otherwise. It is difficult to simplify the above summation since the inner sum does not have a closed form. Figure 2 shows the approximate curve for the density fZ (z). The parameter n0 represents the minimum in-degree considered for computing the random sum. Notice that the sharp peak for the curve n0 = 2 occurs due to the influence of the 2-fold convolution of Figure 1. For higher starting values of n, signifying more densely connected pages the curve becomes more even. Finally, to derive the density fR (r) of a linear function of Zi , we adopt a similar approach as before for finding the density of out-degree inverse, except that the function φ(Zi ) = 1 − d + dZi is strictly increasing. We have from Equation 3 FR (r) = Pr(Ri r) = Pr(1 − d + dZi r) = Pr Zi r−1 d +1 since φ is strictly increasing. Therefore, r−1 = FZ +1 (4) d
Statistical Distribution of a Hyperlink-Based Web Page Quality Metric
25
Random sum 0.35 n0
0.3
>2
0.25 0.2 0.15 n0
0.1
>3 n0
>4 n0
0.05
>5 n0
>6
0 0
2
4
6
8
10
12
Fig. 2. Density of a sum of uniformally distributed random variables. Each curve represents the density function given by ∞ n=n0 fZ|N (z|n)fN (n). For computation purposes we approximated the upper limit to be n = 20.
and differentiating to obtain probability density, d d FR (r) = fZ (φ−1 (r)) (φ−1 (r)) dr dr 1 r−1 = fZ +1 d d
fR (r) =
(5)
Figure 3 shows the unnormalized Pagerank distribution FR (r) denoting the probability Pr(R < r). For n0 = 2, nearly two-thirds pages have a Pagerank within 10% of its total range which means that Pagerank follows a highly concentrated distribution as do Web page in- and out-degrees. This confirms our earlier conjecture that Pagerank distribution is affected by the degree distributions.
3
Experimental Verification
We have conducted experiments to verify the theoretical exercise of the previous section. We summarize our results here. Detailed discussion on the experiments is given in [3]. We do so in two parts—first we verify that Web page out-degrees follow a Lotka distribution and then compare the derived theoretical distribution for Pagerank with the observed distribution of the metric, computed over a given sample of pages. Our experiments required large crawls of the WWW and were undoubtedly limited by the scale of available computing resources, primarily memory and network bandwidth. Our crawls were different from those performed by a typical search engine in that they did not download pages for storage, but merely obtained the url and hyperlink connectivity of visited pages. Web crawls
26
D. Dhyani, S.S. Bhowmick, and W.K. Ng
Pagerank distribution 0.6 n0 = 2
0.5 0.4 0.3
n0 = 3
0.2 0.1 0
0
2
4
6
8
10
12
14
16
18
20
Fig. 3. Derived Pagerank probability distribution FR (r) with parameter d = 0.5 for two values of minimum in-degree n0 .
for our experiments were performed using a Java-based toolkit for developing customized crawlers, known as sphinx (for Site-oriented Processors for html Information) [6]. To test the power law phenomenon which serves as our own starting hypothesis in the theoretical determination of Pagerank distribution, we conducted experiments on a crawl of over 2000 pages. A log-log plot of outdegree distributions is shown in Figure 4. This plot appears linear as expected for a power law distribution. We computed the Pagerank of over 2000 pages (β = 40) with the objective of confirming the theoretically derived distributions of the previous section. Following graph completion, the computation converged with residual error below a threshold of 10−4 . We used a damping factor of 0.5. A log-log frequency plot of Pagerank values suggests that Pageranks also follow a power law distribution. We confirmed this intuition by fitting the observed Pagerank distribution with a power law function shown in Figure 5. Finally, we present in Figure 6 the comparison of the cumulative Pagerank distribution with its theoretically derived counterpart FR (r) from the previous section (see Equation 4). As mentioned earlier, the theoretical expression for Pagerank distribution employs infinite convolutions of uniform densities. For computation purposes, we approximated this with a limit of 20-fold convolution. The derived cumulative distribution was obtained by integrating the area under the density curve of Equation 5 using the trapeziod rule. We see that the observed distribution of Pagerank compares reasonably with the distribution of Equation 4, derived from out-degree distributions.
Statistical Distribution of a Hyperlink-Based Web Page Quality Metric
27
Degree distribution 1000
100 Frequency 10
1
1
10
100
1000
Fig. 4. Log-log plot of out-degree distribution of 2024 Web pages in the NTU domain. Pagerank distribution 50
Observed Fitted
45 40 35 30 Frequency 25 20 15 10 5 0
0
0.002
0.004
0.006
0.008
0.01
Fig. 5. Pagerank distribution for 1889 pages in the NTU domain fitted to the analytical curve y = αx−β where α = 1.7 × 10−10 , β = 3.43. Goodness of fit parameters: χ2 = 47.78, asymptotic standard errors for α and β were 180.9%, 6.84% respectively.
4
Conclusions
In this paper we treated the problem of ascertaining the statistical distribution of some well-known hyperlink based Webpage quality metrics. Based on empirical distributions of Webpage degrees, we derived analytically the probability distribution for the Pagerank metric and found it to follow the familiar inverse polynomial law reported for Webpage degrees. We verified the theoretical exercise with experimental results that suggest a highly concentrated distribution of the metric. Our work on distributions of hyperlink metrics can be extended
28
D. Dhyani, S.S. Bhowmick, and W.K. Ng
Pagerank Distribution 1
observed derived
0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
Fig. 6. Theoretical probability distribution function of Pagerank (computed with 20fold convolution) and the observed normalized cumulative distribution of NTU Pagerank for a crawl of 2026 pages.
by conducting similar exercises for other types of significance metrics for both quality and relevance. It is not clear whether the high concentration depicted by hyperlink metrics is a special consequence of the power law behavior of Webpage degrees or the manifestation of a general regularity in Webpage significance, irrespective of the metric in particular.
References 1. A. Broder, R. Kumar, F. Maghoul, P. Raghavan, S. Rajagopalan, R. Stata, A. Tomkins, J. Wiener. Graph Structure of the Web. Proceedings of the Ninth World Wide Web Conference, Amsterdam, Netherlands, May 2000. 2. S. Brin, L. Page. The Anatomy of a Large-Scale Hypertextual Web Search Engine. Proceedings of the Seventh World Wide Web Conference, Brisbane, Australia, April 1998. 3. D. Dhyani. Measuring the Web: Metrics, Models and Methods. Master’s Dissertation, School of Computer Engineering, Nanyang Technological University, Singapore, 2001. 4. J. Kleinberg. Authoritative Sources in a Hyperlinked Environment. Proceedings of the ACM-SIAM Symposium on Discrete Algorithms, 1998. 5. J. Kleinberg, R. Kumar, P. Raghavan, S. Rajagopalan, A. Tomkins. The Web as a graph: measurements, models, and methods. Proceedings of the Fifth International Conference on Computing and Combinatorics, COCOON , Tokyo, Japan, July 1999. 6. R. Miller, K. Bharat. sphinx: A framework for Creating Personal, Site-specific Web Crawlers. Proceedings of the Seventh International World Wide Web Conference, Brisbane, Australia, April 1998.
MWM: Retrieval and Automatic Presentation of Manual Data for Mobile Terminals Masaki Shikata1 , Akiyo Nadamoto2 , Kazutoshi Sumiya3 , and Katsumi Tanaka3 1
2
Graduate School of Science & Technology, Kobe University, Rokkodai, Nada, Kobe 657-8501, Japan
[email protected] Communication Research Labolatory, Seikatyouhikaridai, Sagara, Kyoto 619-0289, Japan
[email protected] 3 Graduate School of Informatics, Kyoto University, Yoshida Honmachi, Sakyo, Kyoto 606-8501, Japan {sumiya, ktanaka}@i.kyoto-u.ac.jp
Abstract. In this paper, we propose a new way of retrieving and presenting manual data from the Web, based on a state transition diagram and data conversion into synchronized multi-modal contents. The manual data of a target device is assumed to consist of three parts: (1) manual documents in XML, (2) state transition diagrams in XML, and (3) three-dimensional computer graphics (3D CG) of the target device, which are also in XML. The prototype under development is called a Mobile Web Manual (MWM). The major features of MWM are as follows: (a) acquisition of the present state of a target device by wireless communication, (b) retrieval of manual data by finding a path between the present state and a given goal state, (c) automatic generation of 3D animation from the retrieved manual data, and (d) presentation by 3D animation synchronized with synthesized speech.
1
Introduction
Recently, due to the rapid progress of mobile communications technologies, it is possible to use not only the Internet but also a short-distance wireless network, such as Bluetooth[1] and 802.11b. With these mobile communications technologies, mobile terminals can communicate with other types of equipment, acquire the present state of devices, and perform real-time operations. However, not only do most mobile terminals have small displays, but they also do not have full keyboards. This makes it difficult for mobile terminals to display the ordinary, text-based manual data that can be displayed with conventional Web browsers. In other words, mobile terminals impose many restrictions on content presentation and user interactions. We have proposed [2][3] a way of viewing information together with 3D CG (three-dimensional computer graphics) in a
This research is partly supported by the joint research project with NTT DoCoMo, and the research is also partly supported by the Japanese Ministry of Education, Culture, Sports, Science and Technology under Grant-in-Aid for Scientific Research on “New Web Retrieval Services Based on Discovery of Web Semantic Structures”, No. 14019048, and “Multimodal Information Retrieval, Presentation, and Generation of Broadcast Contents for Mobile Environments”, No. 14208036.
R. Cicchetti et al. (Eds.): DEXA 2002, LNCS 2453, pp. 29–39, 2002. c Springer-Verlag Berlin Heidelberg 2002
30
M. Shikata et al.
more automatic, passive style. With our approach, several attributes or description data concerned with a 3D CG object can be viewed passively in mobile terminals, because our approach automatically transforms the attribute data or description data of a 3D CG object into multi-modal synchronized contents. That is, our approach automatically generates 3D CG animation, which is synchronized with speech synthesized from its attribute or description data. A number of research groups are working on controlling smart appliances from a single handheld device based on those mobile terminals, which will be able to communicate with smart appliances using a short-range wireless network. Philips Pronto[4] is the universal remote controls that are available today for controlling many different consumer electronic products. The interface, which is displayed on mobile terminals to control appliance, must be hand-programmed in Pronto’s system. Correspondingly, Jeffrey et al.[5], Abrams et al.[6],the IBM project[7] are working on automatically generating interface, which is displayed on mobile terminals to control appliance. These works are using XML-based language to describe user interface. Many research groups are working on automatic generation of a help or a manual of software. For example, Roberto et al.[8] presented the system that generates hypertextbased help about data presented in application displays, commands to manipulate data, and interaction techniques to invoke commands. Fiorella et al.[9] proposed that explanation by Animated Agent is generated automatically. Our research is in agreement with these researches in respect of automatically generating and representing explanation. Harold Thimbleby et al.[10][11][12] proposed a way of generating user manuals in terms of design integration, so that all aspects of usability (e.g., user interface, help systems, training, documentation) evolve concurrently. He implemented a prototype system called HyperDoc, which animates models of interactive devices, generates user manuals, and supports device analysis. However, few system designers and manual writers use a manual-generating system such as HyperDoc. The feature common to HyperDoc and our proposed approach are the introduction of a finite state machine (FSM) model to describe the behaviors of the target device. HyperDoc does not use XML. In our proposed system, the behavior of the target device is described in XML as a hierarchical state transition diagram (i.e., a state chart) as well as XML-based manual documents. Also, HyperDoc does not take mobile terminals into consideration, but our system does. In this paper, we propose a mechanism of presenting the manual data of device operation using 3D animation and synthesized speech manual text with mobile terminals. We call this mechanism the Mobile Web Manual (MWM). The MWM can relate the state transition, the manual document, and the 3D CG, each of which is written in XML. Usually, a computer operation manual may use text, images, 3D CG, and movies. MWM targets the text and the 3D CG. The left side of Fig.1 shows a system image of the MWM. Users input into the mobile terminals the target operations (goal state) that they want to perform. The mobile terminal can acquire the present state of a device by short-distance wireless communication. The system computes the shortest path of the state transition between the target operations and the present state in the state chart. The shortest path of the state transition can be the action scenario of the 3D animation. The system retrieves the manual from the state
MWM: Retrieval and Automatic Presentation of Manual Data for Mobile Terminals
31
Fig. 1. Mobile Web Manual(MWM) and Correspondence manual and state transition diagram
chart and generates the speech scenario of the 3D animation from the manual. Then, the system creates the 3D animation automatically, based on the action and speech scenarios. The functions of the MWM are as follows: – Get the present state of a smart device by short-distance wireless communication. – Create 3D CG animation automatically by using the state chart, the text manual, and 3D CG, each in XML. – Present the 3D CG animation manual based on the 3D CG animation and synthesized speech. The remainder of this paper is divided as follows: Section 2 discusses the retrieval of the manual data based on the FSM model. Section 3 discusses the presentation of the operation manual on mobile terminals. Section 4 presents a prototype system. Section 5 summarizes our work.
2
Retrieval of Manual Data by FSM Model
In this section, we describe a mechanism to retrieve the necessary portions of the operation manual data using the FSM (finite state machine) model. As stated previously, in our system, the manual data for devices consists of the following three components: – A manual document, written in XML – A state transition diagram, written in XML – 3D CG data of target devices, written in XML In order to retrieve the manual data for user requests, the system uses the current state and the desired state of the target device to construct the scenario of the 3D CG manual. The desired state is computed by the path to the goal state. It is possible for the system
32
M. Shikata et al.
to acquire the current state of the target device with using the short-distance wireless communications network. We will describe the procedure of specifying state transitions. The system retrieves the desired states by user-specified keywords. This is important because some users may not understand the meaning of operation (i.e.,sequence) names. For a set of user-specified keywords and an acquired current state, we need to retrieve all the reachable states whose corresponding text segment contains the user-specified keywords. In the following, we show an algorithm to retrieve all the reachable states. The input to the algorithm includes (1) an XML-based operation manual document, M , (2) a user-specified keyword, key. The output of the algorithm is a list of reachable state names. Let Tkey denotes a text segment that contains the user-specified keyword key. Here, we assume that the manual document has a tree structure that represents the logical structure defined in XML. That is, the manual document forms a tree structure whose vertices are tags, text segments or images (See the upper left portion of Fig.2). Text nodes and image nodes are all leaves of the tree. Step 1 Search the manual document tree for all the text segment nodes each of which contains the user-specified keyword key. Step 2 For each text segment node Tkey containing the keyword key, find the minimal tagged areas (corresponding to a subtree of the manual document tree) that contain both the text segment node Tkey and a non-leaf node with the attribute which is the state or transition. The reason for having applied the minimal tag region is that the neighborhood tags of the sentence that contains user specified keyword are assumed matched with the target state. Here, a tagged area denotes a continuous portion of the manual document, M , which starts at an arbitrary tag X and ends at the corresponding tag X. Step 3 For each minimal tagged area which is found in Step 2, retrieve the corresponding state name in the state chart. Fig.2 shows an illustrative example of the algorithm. The right portion of the figure shows an XML manual document. The upper left portion of the figure shows the XML logical structure of the manual document represented as a tree. The lower portion of the figure shows the correlation between XML tags with the attribute and the state/transition in the state chart. Here, ti denotes a text segment node or an image node. < X∗ > denotes a tag with an attribute which corresponds to a state or a transition in the state chart. < X > denotes a tag that does not correspond to a state or a transition. Sj is an attribute data of < X∗ >, and it denotes a state and j is the state number. Tkl is also an attribute data of < X∗ >, and denotes a transition edge in the state chart. (k is the start state number of the transition, and l is the end state number.) The state or the end state of the transition has the potential to be an item in the output candidate list. The following is an example of the above algorithm for the Fig.2. ti (i = 1, 2, ..., 11) are leaf nodes. In this case, the text node, which includes user specified keyword, is assumed only t10 . Each broken line surrounds the minimal tagged regions of t10 . Each region’s non-leaf node with the attribute, which is the state/transition, becomes < P ∗ > or < Q∗ >. < P ∗ >’s attribute’s value is S3 , and < Q∗ >’s attribute’s is T34 . Then S3 and S4 become the item of the candidate list.
MWM: Retrieval and Automatic Presentation of Manual Data for Mobile Terminals
TAG Construction
t1
t2
t3
t5
<J*>
the tag matched to states or transitions ( <J*>,,,,) the normal tag (,,<W>,) the text node or the image node (t1,..,t10) each broken line surrounds the minimal tagged regions of t10
XML Document (Manual:M)
t1
t2
t3
t6 <W>
t7
t8
t4
t8
t10
t3
t7
<J*>
<W>
t6
t5
t4
33
t9
t11
t9
t10 t11
state chart T12
S2
T34 S4
S3
S1
T43
T21
<J*>
Fig. 2. Correlation between titles and states/transitions
3
Presentation of Operation Manual on Mobile Terminals
Users select the goal state from the obtained candidate list of states, and send it to the MWM server. Then, the shortest path from the current state to the goal state is calculated. The MWM server retrieves the necessary portions of manual data that correspond to the shortest path. This shortest path should be the scenario for constructing the 3D CG manual. Each state or transition attaches surfaces to the 3D CG, plus images and text data as materials. The presentation data is automatically generated and organized based on the scenario and materials (See Fig.3). Generated and organized manual data is presented in a multi-modal manner. That is, our system generates a 3D CG animation synchronized with speech guidance. Using 3D CG enables users to recognize spatial positions and operation directions intuitively. However, it may be easier for some users to understand an operation by using still images or movies. Consequently, the movement from one operation to another operation is presented by a rotating 3D CG model of the devices. The explanation of an operation is presented by a still image that is synchronized with the synthesized speech. Fig.3 shows the flow of the presentation. The presentation data in this paper is as follows: – Speech data: explanations, attentions and annotations, which are transformed from the retrieved text data in an operation manual – 2-D visual data: still images – 3-D visual data: 3D CG model In order to determine the correlation between portions of a manual document and the states/transitions in a state chart, we designed the XML tags and attributes, which are shown in Table 1. These tags are identified with an mwm namespace. These tags and attributes are used in a state chart. We defined the tags and attributes in order to
34
M. Shikata et al.
Fig. 3. Presentation Data Generated and Organized based on Scenario and Materials Table 1. MWM’s tags and attributes in a state transition diagram Element manual sequence
Attribute Specification Root element to describe manual data in state chart. As is the case with < seq > in SMIL, data on and after it is in sequential order parallel As is the case with < par > in SMIL, data on and after it is in parallel order explanation explanation text in speech file manual filename id explanation ID annotation annotation or attention text in speech out. file annotation or attention in speech id annotation ID image an image file represented in explanation file a still-image file name operationface This tag matches state (or transition) to the surface of 3D CG file 3D CG file name face the surface which is used to generate rotation of 3D CG animation
specify the data that should be synchronized with speech guidance. For this purpose, the id attribute is used. The id attribute is added to a tagged area that includes text in the manual document. The text should be transformed into voice speech. However, sometimes only a certain portion of a text segment is transformed into synthesized
MWM: Retrieval and Automatic Presentation of Manual Data for Mobile Terminals
35
speech. Therefore, it is difficult to specify a text portion only by these tags. For this purpose, we introduce the following tags to specify the necessary text portion: – explanation: the region surrounded by this tag denotes the data needed for the voice explanation of operations (transitions) with the attribute explanation ID. – annotation: the region surrounded by this tag denotes the data needed for voice annotation or voice attention of states or operations (transitions) with attribute annotation ID. If tags are used in a still image that has a text caption as its attribute value, then the text caption is transformed into speech data. The goal is to represent an operation manual on a mobile terminal in a multi-modal manner. However, most mobile terminals have a physically narrow display, and they do not have full keyboards. Therefore, mobile terminals impose many restrictions are on content presentation and user interactions. We generate a 3D CG animations based on the state charts. Fortunately, most Web3D[13] representations of 3D CG models are based on XML, and therefore, it is easy to make a correlation between the state chart and the 3D CG model description. MWM makes a correlation between the surfaces of the 3D CG model and the states/transitions of the state chart. In X3D[14], the IndexedFaceSet tag denotes a surface tag. We define the facename attribute for the IndexedFaceSet tag. The facename attribute value provides a unique name for a surface. In a state chart, operationface tag is used to denote the corresponding data of the 3D CG model, and the face attribute is defined for the operationface tag (See Table 1). The face attribute is the facename attribute for the IndexedFaceSet tag. In this way, MWM matches the IndexedFaceSet tag of X3D and the operationface tag of a state chart. Fig.4 is a simple example that illustrates the correlation between a state chart and presentation data. The upper left portion of Fig.4 shows the XML description of a state transition in a state chart. XMI(XML Metadata Interchange)[17], UXF(UML eXchange Format)[18],etc. exist as a format which describes the state chart. In our work, we use UXF to describe a state chart. However our work is independent of specific format, because our purpose is using the ease of the extendibility of XML and applying our system to any XML formats. We describe a state transition from Recording state to Ready state. The lower left portion of this figure shows the presentation flow by the state chart. The upper right portion of the figure shows a description of the 3D CG model in X3D, and the lower right portion shows a fragment of a manual document described in XML. The presentation flow is as follows: 1. First, the 3D CG model is presented. It rotates from its display screen area to the area having a stop button. 2. After the rotation, the text, which is specified for explanation, is transformed into voice speech and is played. After the speech is finished, the stop button turns to the users. 3. After the voice speech is over, the 3D CG model rotates from the stop button to the display area. 4. After the rotation, the device’s still image is presented. At the same time, in a synchronized manner, the text, which is specified as annotation text, is read as voice speech.
36
M. Shikata et al.
Fig. 4. Presentation control by a state chart and manual data
4
Prototype System
In this section, we describe a prototype system based on our MWM. The prototype system was developed by Java and transported into a celluler phone, which is used for FOMA[15] service. (FOMA service is the 3rd-generation cellular-phone service currently offered in Japan by NTT DoCoMo.) Since it is difficult for us to develop a device that supports a FSM and a wireless communications network, we created a 3D CG model as a virtual device on a desktop PC. Also, since a viewer to represent the 3D CG model described in XML on FOMA Platform dose not exist, we utilized i-motion[16] that is service to reproduce specified format video data on a FOMA terminal. That is, 3D CG animation synchronized with synthesized speech guidance, which is changed into the video format for FOMA in advance, and it performed communicating the video of suitable explanation to FOMA. On the MWM server side, we used X3D to represent 3D CG, by which the final CG animation is generated in the form of a VRML file. In order to describe the state charts in XML, we utilized UXF, which is an XML format, to represent UML(Unified Modeling Language). Fig.5 shows the information flow of our
MWM: Retrieval and Automatic Presentation of Manual Data for Mobile Terminals
37
Fig. 5. Information Flow of Our MWM Prototype System
prototype system. The following is one example of possible uses of our MWM prototype system: 1. A user wants to change the state of a device (e.g., video camera), and obtain its current state by using a FOMA terminal. 2. The obtained current state, the product data (e.g., product ID, etc.) and related keywords are transported from the FOMA terminal to the MWM server. 3. The system retrieves manual data based on the current state and the state chart. 4. The user receives a list of the candidates, selects a goal state from this list, and sends the selected goal state to the MWM server. 5. The MWM server automatically generates the 3D CG animation of the retrieved manual data, which is an operation sequence from the current state to a goal state. 6. 3D CG animation, synchronized with synthesized speech guidance, is displayed on the FOMA terminal.(This prototype system represents movie data which is translated from 3D CG animation.)
5
Concluding Remarks
In this paper, we proposed both of a new way of retrieving manual data by a sta te transition diagram and a way of presenting retrieved data in a multi-modal manner by
38
M. Shikata et al.
mobile terminals. The manual data of a target device is assumed to consist of three parts: (1) manual documents in XML, (2) state transition diagrams described in XML, and (3) a 3D CG of the target device. The major features of MWM are as follows: (a) acquisition of the present state of a target device by wireless communication, (b) retrieval of manual data by finding a path between the present state and a given goal state, (c) automatic generation of 3D CG animation from retrieved manual data, and (d) presentation (by 3D CG animation synchronized with synthesized speech). The main contributions of the present paper are summarized as follows: – XML-based representation of the operation manual Operation manual data of devices consist of manual documents, state charts, and the 3D CG of the devices, all in XML. – Manual data retrieval by FSM (state chart) Based on a current state acquired from the devices and a goal state, portions of the manual data are retrieved by finding a shortest path in a state chart. – Automatic generation of 3D CG animation synchronized with synthesized speech Retrieved manual data is automatically transformed into a 3D CG animation synchronized with synthesized speech for the user’s passive viewing.
References 1. Bluetooth Available at http://www.bluetooth.com/ 2. N.Nadamoto, M.Shikata, T.Yabe, K.Tanaka:Autonomous Presentation of 3Dimensional CG Contents and Their Attribute Information, submitted to journal. 3. M.Shikata, A.Nadamoto, K.Tanaka:Retrieval and Reorganization of Operation Manual and Obtain Machine State Information by Mobile Terminals, (in Japanease), Technical Report of IPSJ, Vol.2001, No.70 01-DBS-125(1), pp.343-350, 2001. 4. Philips.Pronto Available at http://www.pronto.philips.com/ 5. J.Nichols, B.A.Myers, M.Higgins, J.Hughes, T.K.Harris K.Harris, R.Rosenfeld, M.Pignol: Generating Remote Control Interfaces for Complex Appliances, Submitted for Publication. 6. M.Abrams, C.Phanouriou, A.L.Batongbacal, S.M.Williams: UIML:An ApplianceIndependent XML User Interface Language, Computer Networks, Vol. 31, 1999, pp. 1695-1708. 7. K.Eustice, T.Lehman, A.Morales, M.C.Muson, S.Edlund, M.GuillenF: A Universal Information Appliance, IBM Systems Journal, 1999, 38(4),pp.575-601. http://www.research.ibm.com/journal/sj/384/eustice.html 8. B.De Carolis, F.de Rosis, S.Pizzutilo: Automated generation of agent behavior from formal models of interaction, in Proceedings of AVI2000, Palermo (Italy), 24-26 May 2000, pp.193199, ACM Press. 9. R.Moriyon, P.Szekely, R.Neches: Automatic generation of help from interface design models. Proceedings of CHI94, Boston, 1994. 10. H.Thimbleby, M.Addison: Intelligent adaptive assistance and its automatic generation, Interacting with Computers, 8(1), pp51-68, 1996. 11. H.Thimbleby: Combining systems and manuals, HCI’93, pp479-488, 1993. 12. H.Thimbleby: Creating User Manuals for Use in Collaborative Design, ACM Conference on Computer-Human Interaction, CHI’96, pp279-280, 1996.
MWM: Retrieval and Automatic Presentation of Manual Data for Mobile Terminals 13. 14. 15. 16. 17. 18.
Web3D Available at http://www.web3d.org/ X3D Available at http://www.x3d.org/ FOMA Available at http://foma.nttdocomo.co.jp/english/ i-motion Available at http://www.nttdocomo.co.jp/english/p s/i/imotion/ XMI Available at http://www.omg.org/technology/documents/formal/xmi.htm UXF Available at http://www.yy.cs.keio.ac.jp/%7Esuzuki/project/uxf/uxf.html
39
Ensuring Recovery for SACReD Web Transactions in the E-commerce Applications 1
Muhammad Younas and Barry Eaglestone
2
1
School of Mathematical and Information Sciences, Coventry University, UK [email protected] 2 Department of Information Studies, University of Sheffield, UK [email protected]
Abstract. This paper presents an extension to a novel Web transaction management protocol, previously defined for a failure-free environment, such that it provides reliable recovery from failure in e-commerce applications. This protocol manages complex Web transactions upon multiple distributed Web services, and overcome limitations of two-phase commit protocols by applying correctness criteria based upon semantic atomicity. Further, it supports enhanced transaction resilience through the use of compensating and alternative transactions. The protocol has been prototyped in a CORBA environment. An evaluation carried out on this prototype shows that the new recovery mechanism minimises the logging cost, increases fault tolerance, and permits independent recovery of autonomous systems.
1. Introduction Limitations of Web-transaction management (W-TM) currently restrict the scope of ecommerce [14]. Specifically, W-TM does not effectively manage complex transactions that aggregate sub-transactions on distributed Web services and data sets. Our contribution is a reliability procedure for managing transactions of this type. Our protocol, which enforces correctness criteria based upon semantic atomicity and transaction resilience (SACReD criteria) [16], has been formally defined and analysed for a failure free environment [17]. Here we extend this work by formally considering the protocol in the presence of failures. Specifications are given in CCS (Calculus of Communicating Systems) notations [8]. Section 2 reviews related work and identifies limitations of current approaches to Web transaction recovery. Section 3, presents a protocol that enforces SACReD criteria for Web transactions in the absence of failures [16, 17], and then extends it by formally specifying a new recovery mechanism which will enforce the SACReD criteria in the presence of failures. Section 4 describes a CORBA-compliant prototype tool that implements the recovery mechanism and the results of an evaluation using it.
R. Cicchetti et al. (Eds.): DEXA 2002, LNCS 2453, pp. 40–49, 2002. Springer-Verlag Berlin Heidelberg 2002
Ensuring Recovery for SACReD Web Transactions in the E-commerce Applications
41
2. Related Work Communications and component system failures on the Web are frequent. However, most W-TM approaches either do not address recovery within the Web environment [11,12,19,20], or rely on two-phase commit (2PC) based protocols to enforce ACID criteria for transaction correctness [1, 5, 10, 3, 7, 6, 2]. Usefulness of 2PC-based solutions is limited by their failure recovery strategy, i.e., aborting the transaction and all subtransactions. This severely affects systems throughput and fault tolerance. Also, performance is impeded by the overhead of logging state changes of Web transactions, and excessive consumption of systems resources. The latter occurs particularly where participating systems cannot communicate with each other to terminate a transaction [17, 13]. A more serious limitation is that, for 2PC to operate, underlying information systems must support either access to prepare-to-commit state or undergo modification to provide this support. The former limits heterogeneity, while the later violates autonomy. Also, in e-commerce application component systems may belong to autonomous businesses, and system modifications for participation in 2PC protocols may not be feasible. Alternatives to 2PC-based W-TM are conversational transactions [11, 12] and Jpernlite [19]. The former comprise autonomous independent transactions, related only in the context of a specific conversation. Jpernlite incorporates extended transactions for Web-enabled software engineering applications (using plug-in codes). Both approaches mainly concern concurrency control mechanisms. The issue of execution guarantees for e-services is raised in [15, 9], but this work has limited scope, since it concerns specific applications, e.g. payment protocols. Also, none of these approaches cover recovery aspects of applications; they do not address systemwide failures of the Web transactions. A concurrency control mechanism for flexible transactions in heterogeneous database environments is proposed in [20], such that alternative subtransaction sets can be specified to execute to compensate or as alternatives to subtransaction within the context of a global transaction. However, simultaneous execution of alternative sets of subtransactions can result in performance degradation [18]. The above limitations motivate our research in which we have defined transaction correctness criteria, which we argue are more appropriate to the Web environment, and protocols to enforce those criteria. Our correctness criteria are based upon semantic atomicity, which requires transactions to commit either all subtransactions, or if any is aborted to reverse, using compensating transactions, the effects of committed subtransactions. This allows subtransactions to unilateral commit and therefore intermediate states of local databases become accessible. This means that consistency is enforced only at component database level, and transactions are not isolated from the effects of others. However, in practice isolation may be undesirable when transactions involve co-operation and negotiation between users. We add resilience as a desirable, but not mandatory property. Web-transactions are vulnerable to failure due to unreliability of the Internet, autonomy of local systems and consequentially increased likelihood of subtransaction failure through system failure or unavailability of requested services, i.e., semantic failure. The resiliency property concerns the capability of transactions to cope with semantic failures, and is therefore an a priori property of a transaction, rather than an a posteriori measure of correctness of its execution. Resilience is increased by associating alternative
42
M. Younas and B. Eaglestone
transactions with subtransactions, to create alternative subtransaction sequences that result in successful completion. Thus, our new criteria ( called SACReD criteria [16]) comprise Semantic Atomicity, Consistency, and Resilience, while Durability remains a requirement, as in ACID criteria. A protocol to enforce SACReD criteria in the absence of failure is formally specified and implemented [17, 18]. Performance evaluation shows improvements over 2PC-based W-TM, in that it is less costly in terms of information logging, while providing relief from system resource blocking. In addition, it increases resilience to systems and semantic failures, while preserving autonomy of participating Web and database servers. In the remainder of this paper we extend the formal definition of the Web Transaction Protocol to recover from failures.
3. Web Transaction Protocol The protocol, P1, defined in this section enforces SACReD criteria in the presence of failures. After establishing basic definitions and assumptions (3.1), the basic protocol procedure, P1, is presented (3.2). P1 is informally specified, (see [17] for its formal specification). The extension to P1 for recovery from failures, is formally defined in 3.3 and 3.4. Section 3.3 covers the failure detection and restart procedures of P1, while 3.4 describes the failure and recovery of P1. 3.1. Basic Definitions and Assumptions A Web transactions, Tweb, is defined as the execution of a Web application which can be divided into well defined units that provide semantically correct transitions between consistent states (with possibly temporarily invalid intermediate states) of shared database systems [17]. Tweb is therefore decomposable into subtransactions, STi which are compensatable if the effects on a database can be semantically undone by a compensating transaction, and replaceable if there is an associated alternative transaction, i.e., a subtransaction can be compensatable-only (CO), replaceable-only (RO), replaceable and compensatable (R-C), or non-compensatable and nonreplaceable (NC-RC). Two types of failure can occur: (i) Component system failures (F2) occur if participating systems experience hardware or software failures. These may halt component systems, thus causing lose of information stored in volatile memory. (ii) Internet failures (F3) give rise to communication failures resulting in lost of messages or delivery failure. As in [13, 18, 1], other types of failures, e.g., improperly ordered messages, failures of CPUs, main memory, and disks, are not considered. The protocol recovers from failure using information relating to previous states of the transaction, written to log files. Logged information includes transaction and subtransaction identifiers, types and status, recorded in the log files at different stages of the protocol. For example, identity of a Web transaction is recorded when it is created. Logging is performed using simple-write or forced-write operations. A simple-write does not suspend the protocol. A forced-write suspends the protocol until information is recorded in the log file. Thus forced-writes significantly affect the performance of the Web transactions.
Ensuring Recovery for SACReD Web Transactions in the E-commerce Applications
43
3.2 The Basic Protocol Procedures Protocol procedures are defined as a multi-process program; WTC (Web transaction coordinator) and each WTA (Web transaction agent) are processes (see 4.). Here we provide informal narrative of their operation. 1.
A new transaction, Tweb, is assigned to a WTC, which records the start of Tweb in a log file using a simple write and initiates the subtransaction, STi. WTC then enters into a wait state, awaiting messages from WTAs concerning completion of subtransactions. (Note, WTC does not send a prepare messages to WTAs, as is necessary in the IYV protocol [4]. Also, WTAs can votes after processing STs, since WTC starts the protocol as soon as it starts the execution of Tweb ).
2.
When a new STi is assigned to a WTA, it records the beginning of STi in the log file using a simple write. After processing STi, WTA sends a commit or abort vote to WTC. If the latter, WTA forcibly records the abort of STi, sends an abort vote to WTC, and declares STi to be local-aborted. Otherwise, WTA forcibly writes a commit decision, sends a commit vote to WTC and awaits WTC’s decision.
3.
WTC receives WTAs’ votes, ignoring any duplicates. If all votes register a commit decision, WTC globally commits, by forcibly writing the commit decision and sending global commit messages to all WTAs. It then terminates Tweb and starts the processing of a new Web transaction.
4.
If any aborted subtransaction, STi, is replaceable, WTC initiates the alternative subtransaction and awaits the WTA’s decision regarding commit or abort of the alternative subtransaction. If STi is not replaceable WTC globally aborts, by forcibly writing the abort decision, sending global abort messages to all WTAs, and terminating the transaction.
5.
After receiving a global commit decision, WTA simply writes the global commit of STi and changes STi’s status from local-committed to global-committed. If STi is locally-committed and WTA receives an abort message, WTA must execute the compensating transaction for STi. This is logged by WTA by simply writing the compensation decision and then marking the end of STi.
3.3. Recovery Mechanism for SACReD Web Transactions We now consider type F2 and F3 failures. Timeout [13,1,17] is used by WTC and WTAs to detect these. It is assumed that an appropriate timeout value is predetermined using the relation: Time-out > DecisionT + MessageT; where DecisionT is the worst case time WTC and WTAs take to reach a commit/abort decision, and MessageT is the worst case time needed to receive a decision. 3.3.1 WTC Restart Process The WTC restart process ensures recovery on the coordinator side, and is designed to respond to the following three situations in which failures of type F2 can occur:
44
M. Younas and B. Eaglestone
Case-1: WTC log file is searched for the commit record of Tweb to establish whether Tweb has committed at the time of failure. This activity is modeled as follow: def
WTC-Restart (Tweb)
= read-log (decision) . WTC-Restart (decision, T /
web
)
def
WTC-Restart/ (decision, Tweb)
= If decision = commit-decision
then send-commit/ (0, no-of-ST) else WTC-Restart// (decision, Tweb) def
send-commit/ (n, no-of-ST)
=
if n = no-of-ST then Global-commit else
no -of -ST
∑
[ g − commit k . send-commit (n+1, no-of-ST) + timeout . send-commit/ (n+1, no-of-ST)]
k =1
def
Global-commit
=
Terminate (Tweb)
In the above, no-of-ST represents the number of subtransactions (∈Tweb). If a commit record is found in WTC log file, then each STi has committed. Thus WTC must inform all WTAs of the global commit decision. WTC then terminates Tweb. Case-2: WTC log file is searched for the abort record of Tweb so as to determine whether Tweb aborted at the time of failure, as follow. def
=
WTC-Restart//(decision,Tweb)
If decision = abort-decision then send-abort/ (0, no-of-ST) else WTC-Restart/// (0, Tweb)
def
send-abort/ (n, no-of-ST)
=
if n = no-of-ST then Global-abort else
no -of -ST
∑
[ g − abort k . send-abort/ (n + 1, no-of-ST) + timeout . send-abort/ (n + 1, no-of-ST)]
k =1
def
Global-abort
=
Terminate (Tweb)
If an abort decision is found in WTC log file, then Tweb was already aborted when failure occurred. WTC must send abort decision to all WTAs, and then terminate Tweb. Case-3: If no global decision is found in the log file then Tweb was active, and therefore there exist one or more outstanding STi (∈Tweb) at WTA sites. Consequently, WTC must communicate with the WTAs to correctly terminate pending STi. To do this, WTC must retrieve each outstanding STi’s status from respective WTAs to determine if they are active, committed, or aborted, and take action accordingly. def
WTC-Restart/// (n, Tweb)
=
if n = no-of-ST then WTC-commit (Tweb)
else (getn (status) . check(n, status, Tweb)) + (timeout . WTC-abort (n, STi, Tweb))
In the above, n is the number of outstanding subtransactions. If WTC can retrieve the status of STi from WTA then WTC proceeds according to check(n, status, Tweb) and takes appropriate action. If WTC cannot retrieve the STi status, then WTA has failed. Thus WTC will timeout for that WTA and will proceed with WTC-abort (n, STi, Tweb) since it will assume STi has aborted.
Ensuring Recovery for SACReD Web Transactions in the E-commerce Applications
45
def
check(n,status,Tweb)
=
if STi-status= local-committed then (local-committedi)) . WTC-Restart/// (n+1, Tweb)
else if STi-status = local-aborted then s - write (local-abortedi)) . WTC-abort (n, STi, Tweb) else if STi-status = active then votei(vote) . check(n, status, Tweb)
If STi has committed, WTC records its commit in the log file and proceeds to the /// next subtransaction by following WTC-Restart (n+1, Tweb). WTC executes WTCcommit (Tweb) provided all STi (∈Tweb) are committed. If any STi is aborted then WTC logs its abort and proceeds according to WTC-abort (n, STi, Tweb). If STi is active WTC must receive the vote from the WTA and check if it is committed or aborted. def
WTC-commit (Tweb)
= f - write (commit-decision) . send-commit (0, no-of-ST) /
def
send-commit/ (n, no-of-ST)
=
if n = no-of-ST then Global-commit else
no -of -ST
∑
[ g − commit k . send-commit/ (n + 1, no-of-ST) + (timeout . send-commit/ (n, no-of-ST))]
k =1
def
Global-commit
=
Terminate (Tweb)
WTC forcibly writes the commit decision to the log file and sends global commit messages to all WTAs. It then terminates the transaction according to Global-commit. WTC-abort (n, STi, Tweb) = if replaceable (STi) then WTC-Restart/// (n, Tweb) else
f - write (abort-decision) . send-abort (0, no-of-ST)
If STi is replaceable, it is replaced by an alternative subtransaction. Otherwise, WTC forcibly writes the abort decision and proceeds according to send-abort(n, noof-ST): def
send-abort (n, no-of-ST)
=
if n = no-of-ST then Global-abort else
no -of -ST
∑
[ g − abort k . send-abort (n+1, no-of-ST) + (timeout . send-abort (n+1, no-of-ST))]
k =1
def
Global-abort
=
Terminate (Tweb)
WTC must then send global-abort messages to all the WTAs, and terminate the transaction. From the above, the following definition of a reliable WTC is deduced. def
Reliable-WTC
=
WTC | (restart(Tweb) + WTC-Restart(Tweb))
46
M. Younas and B. Eaglestone
3.3.2 WTA Restart Process The WTA restart process is designed to respond to four situations that can occur when an F2 failure occurs. Case-1: When WTA restarts its execution, it must determine the status of STi from the log file in the following manner. def
=
WTA-Restart (STi)
read-log (status) . check(STi-status)
def
check (STi-status)
=
if STi-status = committed then local-committed (STi)
/ else if STi-status = aborted then local-aborted (STi) else if STi-status=active then WTA ( STi) else if STi-status = compensated then Terminate (STi)
If STi is committed, then WTA proceeds according to local-committed (STi), /
otherwise it follows local-aborted (STi). If STi is active, WTA follows WTA ( STi). If STi is compensated then WTA needs not to take any action and will terminate STi, as it has already compensated the effects of the locally committed STi. Case-2: In this case, the status of the active subtransactions is determined, whether they are committed or aborted. def
= if g-abort then abort (ST ) else if ST -commit then local-committed (ST )
/ WTA ( STi)
i
i
i
else local-aborted (STi) def
abort (STi)
=
abort-STi . Terminate (STi)
If WTA receives a global-abort message from WTC amid the processing of STi then it will abort and terminate STi. This arises when WTC receives an abort vote for a non-replaceable subtransaction. It must then send global abort messages to all WTAs. Consequently WTAs abort all active subtransactions. If STi is committed then WTA must follow local-committed(STi), otherwise WTA will follow local-aborted (STi). Case-3: This case concerns the situation when STi is locally aborted. def
local-aborted (STi)
= f - write
(abort-decision) . vote i(abort) . Terminate ( STi)
WTA forcibly writes the abort of STi in a log file, sends abort vote to WTC, and declares STi as local-aborted. It then terminates STi. Case-4: This case concerns the situation when a subtransaction, STi, is locally committed. def
local-committed (STi)
= f - write (commit-decision) .
vote i (commit) . Wait (WTC-decision))
Ensuring Recovery for SACReD Web Transactions in the E-commerce Applications
47
WTA sends commit vote to WTC (if it has not sent before), and then waits for WTC’s decision. But before sending the commit decision, WTA forcibly writes commit decision in the log file. def
Wait (WTC-decision)
= get(WTC-decision) . (global-commit . global-commit (ST ))+(global-abort. i
global-abort (STi))
In a wait state, WTA contacts WTC to query the global commit or abort decision. If WTA receives a global-commit message from WTC, it acts according to globalcommit (STi), otherwise it proceeds with global-abort (STi). def
global-commit (STi)
= s - write . (STi-global-commit) .
Terminate (STi)
WTA simply writes the global commit for STi and changes the status of STi from local-committed to global-committed in the log file. WTA then terminates STi. def
global-abort(STi)
=
compensate(STi) .
s - write (ST -compensated) . i
Terminate (STi)
WTA first executes the compensating transaction for the committed STi. WTA then simply writes the compensation decision in the log file, and marks the end of STi. The following definition of a reliable WTA is deduced from the above algorithm. Reliable-WTA = WTA | (restart(STi) + WTA-Restart (STi))
3.4. Failures and Recovery We now describe P1 (section 3.2) in the presence of failures of types F2 and F3. WTC passes through the states: initial, wait, and decision. When WTC recovers from failures in the initial state, it must restart Tweb from the beginning, as no information is logged about the beginning of Tweb. If there is no failure then WTC changes to a wait state. 1.
In wait state, WTC waits for the WTA votes. If any vote fails to arrive due to F3, WTC timeouts and assume STi has aborted. WTC will interpret this as a WTA site failure. WTC aborts STi if it receives an abort vote or if it times out due to lack of a response from WTA.
2.
After recovery in the wait state, WTC executes the restart process, WTC-Restart process (see 3.3.1), and waits for votes from pending WTAs. If no failure occurs, it will commit or abort Tweb, as appropriate.
3.
If WTC fails after having written the global commit or abort decision, it re-sends the decision messages to all WTAs, using WTC-Restart process (see 3.3.1) as some messages may have been received by the WTAs, whereas others may not.
WTA passes through the following states: initial, processing, voting, wait. The behaviour of WTA in these states when errors of type F2 and F3 occur, is as follow: 1.
If WTA fails in the initial state, it must re-execute STi after recovering from failure. Since STi has not yet started, no information is available in the log file. In the case of no failure, WTA records the beginning of STi and starts processing.
48
M. Younas and B. Eaglestone
2.
If WTA fails during the processing of STi, it has to follow the WTA-Restart process (see 3.3.2), which checks the status of STi from the log file. If failure does not occur, WTA changes to the voting state.
3.
If WTA fails after writing the local commit or abort decision in the log file, but has not sent its vote to WTC, the following must happen. After recovery, WTA will follow the WTA-Restart process, in which case, either WTA sends an abort vote to the WTC and terminate STi, or it sends a commit vote to WTC and waits for WTC’s decision concerning global commit or abort.
4.
If WTA fails while waiting for the WTC decision, the following must occur. After recovery, WTA will query WTC to determine that global decision. This is done by following the WTA-Restart process, because it is possible that WTC might have sent the decision while WTA was unable to operate due to failure.
Note that the correctness of the recovery procedure (presented in 3.3 and 3.4) is formally verified in [18], but we are unable to present that due to page limit.
4. Summary and Discussion We have defined a W-TM protocol [17] for reliable recovery of complex transaction from system and communication failures. Novel features are the application of the SACReD correctness criteria which we argue are more suited to the Web environment than the conventional ACID criteria [16], and recovery from failures. This protocol has been validated and evaluated through the development of a CORBA-compliant prototype. This comprises two parts: Web transaction coordinator (WTC), and Web transaction agent (WTA). Both are implemented as CORBA objects, which may be located at different nodes of the Internet. The commit/abort of the Web transaction and its subtransactions, and occurrence of the site failures, are simulated using probability approach. Evaluation of the protocol using the prototype demonstrates performance improvements. Logging costs were reduced by requiring fewer forced writes to the log file. These affects performance, since execution of the protocol must pause while information is logged. Our protocol uses n + 1 forced-write operations to process a Web transaction; where ‘n’ is the number of the subtransactions, compared to 2n + 1 required by 2PC-based protocols. Resilience of a W-TM system is increased by use of alternative transactions. Our evaluation shows that if there exist 0.3 (30%) alternative transactions per Web transaction, the commit probability of a Web transaction increases from (i) 0.9 to 0.927 (ii) 0.8 to 0.848, and so on (see [18] for details). The recovery mechanism does not cause cascading restarts of component systems. Instead, restart processes are executed independently, though they must exchange messages after independently recovering from failures. Ensuring independent recovery is important in e-commerce applications, because component systems providing crossorganizational e-commerce services are largely autonomous [11, 14]. Finally, W-TM systems independently recover from failures of participating systems, and do not require modification to participate in the protocol. Thus autonomy of component systems is maintained.
Ensuring Recovery for SACReD Web Transactions in the E-commerce Applications
49
References 1. 2. 3. 4. 5. 6. 7. 8. 9. 10. 11. 12. 13. 14. 15. 16. 17. 18. 19. 20.
D. Billard: Transactional Services for the Internet. Proc. of Int. Workshop on Web and Database (WebDB’98), Valencia, Spain, March 27-28, 1998. Sylvanus A. Ehikioya, K. Barker: A Formal Specification Strategy for Electronic Commerce. IDEAS, Montreal, Canada, August, 1997. K. Evans, J. Klein, J. Lyon: Transaction Internet Protocol: Requirements and Supplemental Information. Internet-Draft, October 1997. Yousef J. Al-Houmaily, P.K. Chrysanthis: Two-Phase Commit in Gigabit-Networked Distributed Databases. 8th Int. Conf. on Parallel & Dist. Comp. Sys. Sept., 1995. J. Lyon, K. Evans, J. Klein: Transaction Internet Protocol: Version 3.0. Internet-Draft, April 1998 (http://www.ietf.org/ids.by.wg/tip.html) M.C. Little, S.K. Shrivastava, S.J. Caughey, D.B. Ingham: Constructing Reliable Web Applications using Atomic Actions. 6th Int. WWW Conf., USA, April, 1997. M.C. Little, S.K. Shrivastava: Java Transactions for the Internet. Proc. of 4th USENIX Conference on O-O Technologies and Systems, April 1998. Rubin Milner: Communication and Concurrency. C.A.R. Hoare Series Editor, Prentice Hall, International Series in Computer Science, 1989. rd C. P. Martin, K. Ramamritham: Guaranteeing Recoverability in Electronic Commerce. 3 International Workshop on Advanced Issues of E-Commerce and Web-based Information Systems, 21-22 June, 2001, San Jose, California. OMG: CORBAservices: Common Object Service Specification. November 1997 (http://www.omg.org/corba/csindx.htm) J.Ouyang, A.Sahai, V. Machiraju: An Optimistic Commit Protocol for Conversational Transactions. Technical Report HPL-2001-02, Hewlett-Packard Labs, 2001 J. Ouyang, A. Sahai, V. Machiraju: An approach to Optimistic Commit and Transparent Compensation for E-Service Transactions. 14th International Conf. on Parallel and Distributed Computing Systems (PDCS 2001), Dallas TX T.Ozsu, P. Valduriez: Principles of Distributed Database Systems. Prentice-Hall, 1991 th H. Schuldt, A. Popovici: Transactions and Electronic Commerce. 8 Int. Workshop on Foundations of Models and Languages for Data and Objects: Transactions and Database Dynamics, Schloss Dagstuhl, Germany , 1999, Springer LNCS No.1773. J.D. Tygar: Atomicity versus Anonymity: Distributed Transactions for Electronic th Commerce. Proceeding of 24 International Conference on Very Large Databases (VLDB), August 24-27, 1998, New York City M. Younas, B. Eagelstone, R. Holton: A Review of Multidatabase Transactions on the Web: From the ACID to the SACReD. British National Conf. on Databases (BNCOD), Exeter, UK, July 3-5, Springer LNCS, 2000. M. Younas, B. Eagelstone, R. Holton: A Formal Treatment of a SACReD Protocol for Multidatabase Web Transactions. 11th International Conf., DEXA2000, Greenwich, London, 5-8 September, 2000, Springer LNCS, 2000 M. Younas: Web Transaction Management for Multidatabase Applications. Ph.D. Thesis, University of Sheffield, UK, 2001. J.Yang, G.E.Kaiser: JPernLite: An Extensible Transaction Server for the World Wide Web. IEEE TKDE, 1999. A. Zhang, M. Nodine, B. Bhargava: Global Scheduling for Flexible Transactions in Heterogeneous Distributed Database Systems. IEEE TKDE 13(3), 2001, 439-450
Workflow Simulation across Multiple Workflow Domains Xiaohui Zhao Department of Computer Science Harbin Institute of Technology, P.R. China [email protected]
Abstract. In the scenario of multiple homogenous intra enterprise workflow domains, several partial processes cooperate according to certain interoperability models. In this case, workflow simulation requires more flexible and powerful mathematical models. In this paper, BCMP queuing network is imported to workflow based business process simulation filed for the first time. With its special multi-class and multi-chain characteristics, we can provide a workflow simulation across multiple domains using three basic interoperability models. This simulation approach accords well with the dynamic and randomness of workflow. Thus, we can report some performance indexes and analysis before deploying.
1 Introduction 1.1 Introduction Workflow is the computerized facilitation or automation of a business process, in whole or part. Workflow management system (WfMS) uses one or more functional components to perform business activities, to interact with workflow participants (persons or applications), to manipulate workflow process instances’ implementation and supervise the runtime state of process instance [1-3]. Traditional workflow system requires each workflow model to be instantiated first to analyze before applied in practice. However, with workflow simulation, we can predict its performance before deploying. In this paper, a simulation method, based on BCMP queuing network, is proposed, which supports the chained, nested and parallel synchronized interoperability models across multiple workflow domains.
1.2 Related Works There are many researches about integrating workflow representation and simulation/performance analysis together, such as the following: ARIS (Architecture of Integrated Information System), a process-oriented model architecture proposed by Dr. Scheer of University Saarland, which defines four R. Cicchetti et al. (Eds.): DEXA 2002, LNCS 2453, pp. 50–59, 2002. Springer-Verlag Berlin Heidelberg 2002
Workflow Simulation across Multiple Workflow Domains
51
views, i.e. function/process view, organization view, data view and control view [4,5]; GRAI (Graphical Results and Activities Interrelated), a method to describe and analyze production management system, proposed by GRAI lab of University Bordeaux [6]; QSGM (Qualitative Simulation Graph Methodology), developed by Ingalls, et al, a general purposes qualitative discrete-event simulation (QDES) framework that can be well suited to address the PERT (Performance Evaluation Review Technology) scheduling with resource problem [7,8]; A performance analysis approach to very large workflow systems, proposed by Dr. Kwang-Hoon Kim, based on MOL (Method of Layers) in order to evaluate performance of possible spectrum of workflow architectures proposed in the framework [9,10].
1.3 Compare with Petri Net Aiming at model's organizational structure and dynamic behavior, Petri net is easy to represent the change condition and post change state, especially when timed and Stochastic Petri nets come forth. Anyway, it is widely accepted that Petri net is good at distributed, parallel, synchronized system analysis [11-13]. However, Petri net is unfit to represent the change of data value or attributes. In the course of large, complex system modeling, Petri net always encounters the problem of too large state space, which blocks its application area very much. Compared to Petri net, queuing network is specific at having product form solutions. This point can simplify the solution computing a lot, which is much valuable in practice. In addition, the hybrid queuing network may represent the characteristics of both transaction handling and batch workload [14,15].
2 Multiple Domain Design Based on Workflow Interoperability 2.1 Design of Multiple Workflow Domains During the process of intra enterprise information revolution, such as a workflow based ERP (Enterprise Resource Planning) project to a manufacturing enterprise, the functional parts of the ERP system can be separated into several workflow domains according to the business logic relationship and business contact frequency. Such workflow domain is relatively independent and autonomous, each domain only manage the business that contacts with functional parts in the domain. This new design approach is apt to the changing environment, and is more flexible and reliable than the original long life cycle management. Each workflow domain can be represented as a tri-tuple Domain•( Eng, PD, FC ) PD={Process Definition1, … , Process Definition n} ‘PD’ is a set of process definitions in this workflow domain. Process Definition = (BA, BR)
52
X. Zhao
Each Process Definition contains two sets – ‘BA’ and ‘BR’, which denote the set of business activities involved in a business process and the set of the business rules among these activities respectively. FC={Functional Component1, … , Functional Component n} ‘FC’ in is the set of functional components, charged by the engine of the domain, which can deal with certain business activities. The concept of functional component is extended to denote the executor of one or more activities defined in set ‘BA’. Moreover, it can be not only a software system, but also a person or a combination of both. Functional components belonged to the same workflow domain can interpret the same workflow process definition, and share workflow control data and process instance. ‘Eng’, representing the special workflow engine in the domain, is up to interpret the process definitions in set ‘PD’, and deal the corresponding business process by invoking functional components in set ‘FC’. Because various business processes may involve some certain activities, the executors – the functional components may appear in the intersection of proper domains. Fig. 1 shows a typical domain design to ERP system. The detailed dividing algorithm can be found in reference [16].
Workshop Control Quality Management
PDM
Inventory Management
MRP/CRP
Purchasing Management Order Management
DRP Distribution Management
Cost Control Finance Management
TQC
Project Scheduling
Cost Planning
Cost Analysis Total Cost Accounting
R&D TrialManufacture
Fig. 1. Petal-like multiple domain design
The workflow engines of these domains have the same structure and running mechanism. They may use the same modeling language, and all the functional components can be built on the same development platform, such as CORBA or J2EE. However, one domain can only deal with a specific set of processes.
Workflow Simulation across Multiple Workflow Domains
53
2.2 Interoperability Models There are three kinds of classical interoperability models that support the interoperability across workflow domains, i.e. chained service, nested sub process, chained service and parallel-synchronized model [17,18]. Chained Service Model allows one process instance to be joined to another. The two joint nodes can be not only the start and end nodes, but also any two nodes of the process instance. Fig.2a shows such a simple example. This model supports one workflow engine to transfer a process instance to another engine of other domain, then the receiver can deal the process instance itself, and the two engines need no longer to be synchronized. In Nested Sub Processes Model, as Fig.2b shows, a process instance running in a workflow domain just acts as a sub process of one process of another workflow domain. Parallel Synchronized Model (as Fig.2c shows) allows two processes to operate essentially independently, possibly across separate domains, but require that synchronization points exist between the two processes. Synchronization requires that once the processes each reach a predefined point within their respective execution sequences, a common event be generated. However, we do not encounter such cases frequently, because such cases are always tight-coupled ones, most of which have already been gathered into one proper domain. Domain A Process A1 a1
a2
Domain B
Domain A
Domain B
Process B1
Process A1
Process B1
a3
b2
b3
b1 a5
a4
b5
b4
a1
a2
a3
b2
Domain A
b3
b1 a5
a
a4
b5
b4
b
Domain B
Process A1 a1
a2
Process B1
a3
b2
b3
b5
b4
b1 a5
a4
c
Fig. 2. Interoperability models
3 BCMP Queuing Network 3.1 Introduction to BCMP Queuing Network Queuing networks are widely used to analyze the performance of multi-programmed computer systems. The theory dates back to the 1950s, when Jackson first published an analysis of a multiple device system [19]. In 1967, Gordon and Newell simplified the notational structure of these results [20]. Baskett, et al. extended the results to include different queuing disciplines and multiple classes of jobs [21], and they also named it with the capitals of their names, i.e. BCMP network. In this paper, BCMP network is imported to workflow based business process simulation field for the first time. Moreover, Table 1. shows the similarities between the BCMP and workflow.
54
X. Zhao Table 1. Similarities between BCMP network and workflow
BCMP Task class may change during marching period. Task direction may change during marching period. Tasks of multi classes may be in one network. Each node has stochastic service time. Tasks get to each node stochastically. Each node has a task queue.
Workflow A process instance may be transferred to be an instance of another process under the scenario of interoperability. Process definition maybe modified dynamically during implementation period. Processes of multi definitions are always implementing simultaneously. Activity service time may change with different process. Dynamic workflow may add or delete some activities temporally. Each functional component has a work item queue.
Anyway, all of the above use the theory of stochastic process, which begins with the following Stochastic Hypothesis: The behavior of the real system during a given period of time is characterized by the probability distributions of a stochastic process. However, it should be mentioned that Stochastic Hypothesis is an assertion about the causes underlying the behavior of a real system. It can never be established beyond doubt through any measurement.
3.2 Mapping to Workflow First of all, it should be announced that all to be discussed in this section depends on the following assumptions: Stochastic Hypothesis; Each queue has infinite capacity; Each component follows the service rule of First Come First Served (FCFS); Each functional component can only deal one transaction at one time; Task arrivals as a Poisson process; As Fig.3 shows, the processes across workflow domains can be mapped to a BCMP network as a directed graph, which consists of a set of service centers and transitions. Each service center comprises a task queue and a functional component, which is to perform some business activities. A process instance coming to this network will go through it along a certain route. Different from single class queuing network, BCMP network supports multiple classes of tasks, such as instances of engineering process and instances of PDM process. Furthermore, a certain task may change its original route, e.g. an Engineering process instance may go back to do shape design or contrail design again instead of the next activity, when it finishes the performance analysis. Thus, it is called a multichain network, as each task may own more than one route. An instance of a certain process r (r=1,2, ... , R), goes through a network with M functional components, which are marked as 1,2, ... , M. The transition probability of each component is defined in set {qir ; js } . qir ; js denotes that the probability that an instance goes from component i to j with its process definition changing from r to s.
Workflow Simulation across Multiple Workflow Domains
55
Two processes of r and s are interchangeable if an instance of process r can change into another process s, while one of process s can also do this adversely. According to this point, such process definitions are collected into one same subset, if they are interchangeable. As to these subsets, we define the term of routing chain as the following: {(i,r)| component i can be accessed by instances of process r, r• P} P is a subset derived from the mentioned method. Thus, a routing chain contains the information about all components that can be accessed by a certain process. In Fig.3, there are two routing chains, which are connected by solid and hollow arrows respectively. Let eir denotes the times that functional component i is accessed by the instances of process r in routing chain c. As to component i and process r,
∑
( j , s )∈c
(i , r ) ∈ c , we have,
e js q js ;ir + q0;ir = eir
(1)
q0;ir is the probability that an instance comes from external to component i and changes to process r. As to component i, we define µir denotes component i’s service speed to process r. for each FCFS component, it has the same
µir to every process r;
There is another representation form to describe the communication traffic in the network –using the “visit ratio”. Define visit ratio, vir , means the average visit times of an instance of process r during its staying period, and there are
γ r instances come
to the network from external. Then we have,
vir =
eir γr
(2)
The detailed mathematical justification can be found in reference [21]. Divide c by d, we get
vir = q0;ir +
∑
( j , s )∈c
v js q js ;ir
(3)
Define Dir means the total service time of component i to one instance of process r, then we get Dir =
Dir =
∑
e js . In addition,
( j , s )∈c µ js
Dir can also be represented as,
vir ρir , which is equivalent to ρ = γ D . At last, we get = ir r ir µ ir γ r
ρir = γ r
∑
( j , s )∈c
e js
µ js
and then, we can get some important performance parameters, such as
(4)
56
X. Zhao
ρi = ∑ ρir
(5)
c
li =
ρi 1 − ρi
(6)
ρi i =1 1 − ρi M
L=∑
(7)
ρi denotes the time utilization of component i. li denotes the average queue length of component i. L denotes the total queue length of the network. Cost Control and Accounting Processes in Cost Management Domain m5
m4
1-m5 CoS 1-m4
m3
CoC m6
1-m6 1-m3
Engineering Process in Engineering Domain
CoD
CoA
1-m6
1 a1
1
PA
CC
CoP 1 PDM Process in Manufacturing Domain
1 OUT
CA
b1
a2
a3
WCA 1
1
m1
1
SD
b2
EXP
BOP
DM
IN m2
1-m1
1 CD
1
TM
1
1 PC
BOM
1-m2
Fig. 3. Workflow BCMP network. Table 2. Acronyms and Activities in Fig.3. • 1
Abbr
Activity
SD
Shape Design
2 3
CD TM
4
EXP
Contrail Design Trial Manufacturing Experiment
5 6
CC PA
Cost Control Performance Analysis
• 7
Abbr
Activity
DM
8 9
PC BOM
Drawingsheet Management Part Coding Bill of Material
• 13
Abbr
Activity
CoD
14 15
CoA CoP
Cost Decomposition Cost Analysis Cost Planning
10
BOP
Bill of Process
16
CoC
11 12
CA WCA
Cost Accounting Work Center Design
17
CoS
Cost Computation Cost Solution
Workflow Simulation across Multiple Workflow Domains
57
4 Simulation 4.1 Simulation In Fig.3, the numbers, above the arcs, denote the route probability. The cost control process acts as a sub process of the engineering process, and so does the cost accounting and PDM (Production Data Management) processes. In addition, the engineering and PDM processes follow the chained service mechanism. Anyway, the service time of some activities vary with the complexity of practical process instances, which doesn't permit the Stochastic Hypothesis. In this case, we apply an alternative method, which is to add some self-pointing arcs (represented in the dashed) to convert time chanciness into route chanciness. The simulation consists of two steps: Firstly, determine the basic parameters, i.e. {qir ; js } , {γ r } and {µir } ;
{µir } can be gotten from past experience, {γ r } depends on the input speed and {qir ; js } . As to {qir ; js } , besides experience, we need to generate some dummy process instances, which are initiated only with the transition information and corresponding parameters. These dummy instances are driven through the network under the workflow transition rules. Thus we can determine the routing probabilities. Secondly, compute the performance parameters using the mentioned formulas;
4.2 Result Analysis Ten random generated transactions are started every time, and we assume the transaction interval defers to hypo dispersion of [5,15] days, based on the experience from ETO (Engineer to Order) type enterprise. The gray bars in Fig.4a show time utilization of activities in the three processes. The numbers under the horizon axis represent these activities, and the mapping relationship is listed in table 2. From Fig.4a, we can see that cost analysis activity’s utilization is relatively higher than other cost activities. The same conclusion can be got in Fig.4b, which shows task queue length of each activity. The cost control activity’s task queue is relatively longer, which means it may defer the whole business process’s implementation time. Then an estimated conclusion can be drawn that cost analysis activity may be the bottleneck of the whole flow. After a careful study, we can see that typical cost analysis contains five steps, i.e. create activity model, collect cost data, classify cost data, trace to activity and cost analysis. A redesign scheme is to removing the first two steps to the activities of cost decomposition and cost accounting. The black bars in Fig.4ab show the utilization and task queue length after modification respectively. From these two figures, we can see that cost analysis activity’s burden has been reduced to some extent, and so do the activities of cost control process.
58
X. Zhao
Most of the other activities’ utilizations increase, which may be caused by the shortened life cycle of the whole business process. And the task queue length of each activity tends to be balanced. What’s surprising is the mean task sum (as Table.3 shows) still falls when the life cycle has been shortened after the modification. 50
0.8
45 0.7 40 0.6 35 0.5
30
25
0.4
20 0.3 15 0.2 10 0.1 5
0 0
0 1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
0
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
b
a
Fig. 4. Simulation results Table 3. Mean task sum
Mean task sum
Before modification
After modification
2.6180
2.4778
5 Conclusion By mapping workflow functional components to BCMP network nodes, business process flow acts as a series of activities upon this network. Thus, some valuable performance indexes can be computed from this simulation, such as resource utilization, task queue length and etc. Then we can find out the bottleneck factors of original business process, and to test various modified schemes to refine the best solution. Therefore, a closed loop, from process design, evaluation to redesign, has been built up. Likewise, this does contribution to system design and business process reengineering. The short point is that it mainly concentrates on the route selections, while other important aspects, e.g. human factors, are not considered very much.
Workflow Simulation across Multiple Workflow Domains
59
References 1. G.Alonso, D.Agrawal, A.El Abbadi and C.mohan: Functionality and Limitations of Current Workflow Management systems, IEEE Expert, Vol. 12 (1997), pp: 0-29. 2. WfMC. Workflow Management Coalition Terminology and Glossary (WfMC-TC-1011). Technical Report, Workflow Management Coalition. 3. WfMC. Workflow Management Coalition The Workflow Reference Model (WfMC-TC1003). Technical Report, Workflow Management Coalition. 4. Scheer, A.W.: Architecture of Integrated Information Systems - Foundations of Enterprise Modeling, Springer-Verlag Berlin, Heidelberg, (1992). nd 5. Scheer, A.W.: ARIS Business Process Frameworks, 2 Edition, Springer, Berlin. 6. Doumemeingts G.: GRAI approach to designing and controlling advanced manufacturing systems in CIM environment. In Advanced Information Technologies for Industrial Material Flow Systems, New York; Springer, (1989). 7. Ingalls, R.G.; Morrice, D.J.: PERT scheduling with resources using qualitative simulation graphs Simulation Conference, 2000. Proceedings. Winter , Volume: 1 (2000). pp:362370. 8. Ingalls, R. G., D. J. Morrice, A. B. Whinston: Implementation of Temporal Intervals in Qualitative Simulation Graphs. ACM Tracnsactions on Modeling and computer Simulation (2000). 9. Kwang-Hoon Kim; Dong-Soo Han: Performance and scalability analysis on client-server workflow architecture. ICPADS 2001. Proceedings. Eighth International Conference on Parallel and Distributed Systems (2001) pp: 179 –186. 10. Kwang-Hoon Kim, Clarence A.Ellis: Workflow performance and scalability analysis using the layered queuing modeling methodology" in Proceedings of the 2001 International ACM SIGGROUP Conference on Supporting Group Work - Volume 2001 September (2001). 11. Petri C A.: Communication with Automata, New York: Griffiss Air Force Base. Tech. Rep. RADC-TR-65-377, 1(1), (1996). 12. Feldbrugge F, Jensen K.: Petri net tool overview (1986). In: LNCS, 1987, 255: 20-61 13. Best and Fernandez C.: Notations and terminology on Petri net theory. Petri Net Newsletters 23, (1986), pp: 21-46. 14. P.J. Denning and J.P.Buzen: The Operational Analysis of Queuing Network Models, ACM Computing Survey, Vol. 10, No.3, Sep. (1978), pp:225-261. 15. E.D.Lazowska, et al.: Quantitative System Performance, Computing System Analysis Using Queuing Network Models, (http://www.cs.washington. edu/homes/lazowska/ qsp/). 16. Xiaohui Zhao: Process Integration based on Interoperability across Multiple workflow Domains (submitted to publish). 17. C. Bussler: Enterprise-Wide Workflow Management. IEEE Concurrency Journal, July, (1999), pp: 94-100. 18. WfMC. Workflow Management Coalition Reference Model Interface 4 Workflow Interoperability The Key to E-Commerce and to process, Workflow Management Coalition. 19. Jackson, J. R.: Networks of Waiting Lines. Oper. Res. 5 (1957), pp: 518-521. 20. Gordon, W. J.; Newell, G. F.: Closed Queuing Systems with Exponential Queuing Network Familiy. Acta Inf. 7.2 (1976) pp: 123-136. 21. F.Baskett, K.M.Chandy, R.R.Muntz, F.Palacios-Gomez: Open, Closed and Mixed Networks of Queues with Different Classes of Customers, J.ACM, vol.22, No.2, (1975), pp: 248-260.
Transactional Workflows or Workflow Transactions? Paul Grefen Computer Science Department, University of Twente P.O. Box 217, 7500 AE Enschede, Netherlands http://www.cs.utwente.nl/~grefen Abstract. Workflows have generally been accepted as a means to model and support processes in complex organizations. The fact that these processes require robustness and clear semantics has generally been observed and has lead to the combination of workflow and transaction concepts. Many variations on this combination exist, leading to many approaches to transactional workflow support. No clear classification of these approaches has been developed, however, resulting in a badly understood field. To deal with this problem, we describe a clear taxonomy of transactional workflow models, based on the relation between workflow and transaction concepts. We show that the classes in the taxonomy can directly be related to specification language and architecture types for workflow and transaction management systems. We compare the classes with respect to their characteristics and place existing approaches in the taxonomy – thus offering a basis for analysis of transactional workflow support.
1 Introduction Workflows have generally been accepted as a paradigm for modeling and supporting processes in complex organizations. Often workflow processes have a business character, but workflow concepts have also been used for other processes types, e.g., scientific processes or software production processes. The use of workflows for core processes of organizations has lead to the requirements of clear process semantics and robustness in process execution, both in regular process execution and under exception or error conditions. The notions of transaction management, already used for several decades in the database world, have been combined with workflow notions to satisfy these requirements. Resulting from this, the notion of transactional workflow or workflow transaction has emerged. Many variations on the notion of transactional workflow or workflow transaction have been developed, however, by merging the worlds of workflow and transaction management in different ways – the two more or less synonymous terms are an omen of this. No clear classification has been developed yet that provides a framework for the analysis of transactional workflow models and systems supporting these models. Matching models and systems with application requirements and comparing approaches is therefore not easy. In this paper, we present a classification framework that provides two main classes for the combination of workflows and transactions, based on the relation between workflow and transaction concepts. The main classes are further refined into subclasses with specific properties, resulting in six basic classes. We show that the conceptual classes can directly be mapped to specification language and architecture classes for workflow and transaction management support. We analyze the classes with respect to their goal, means to achieve this goal, and advantages and disadvantages. The R. Cicchetti et al. (Eds.): DEXA 2002, LNCS 2453, pp. 60–69, 2002. c Springer-Verlag Berlin Heidelberg 2002
Transactional Workflows or Workflow Transactions?
61
Transactional Workflow
SEP
WF/TR TR/WF TR+WF TRWF
INT
WF
TR
Figure 1. Transactional workflow taxonomy
framework and analysis together provide a clear basis for comparing approaches and selecting specific approaches for specific application classes. The structure of this paper is as follows. In Section 2, we describe our basic taxonomy that underlies the classification presented in this paper. In Section 3, we present the conceptual point of view of our classification, focused around language aspects. Section 4 presents the system point of view, centered on architecture aspects. In Section 5, we apply the framework by comparing the various classes and classifying existing approaches to transactional workflows. Section 6 contains conclusions.
2 The Taxonomy To support transactional workflows, there are two basic approaches: either transactional aspects and workflow aspects are treated as separate issues, or they are seen as one integrated issue. In the former case, separate transaction and workflow models exist that are combined to obtain transactional workflows. In the latter case, one single transactional workflow model is used. These two main classes are refined below. In the situation where we have separate workflow and transaction models, we need to relate these two models. We have three possible basic relations, based on the abstraction relation between the models: Workflows over transactions (WF/TR): workflows are more abstract than transactions – transaction models are used to provide semantics to workflow models. Transactions over workflows (TR/WF): transactions are more abstract than workflows – workflow models are used to provide process structure to transaction models. Transactions and workflows as peers (TR+WF): workflow and transaction models exist at the same abstraction level – workflow and transaction models can be seen as two submodels of an implicit, loosely coupled process model. In the case of one single model for both workflow and transaction aspects, obviously there is no relation between models. There are, however, three main variants with respect to the nature of the single model: Hybrid transactional workflow model (TRWF): a single hybrid model is used that contains both transaction and workflow concepts. Transactions in workflows (WF): a single workflow model is used, in which transactional aspects are mapped to workflow primitives.
62
P. Grefen
Workflows in transactions (TR): a single transaction model is used, in which workflow aspects are mapped to transaction primitives. The resulting taxonomy is depicted in Figure 1, in which the ‘SEP’ main class contains basic classes with separate models for workflows and transactions, the ‘INT’ main class contains basic classes with a single integrated model. We use this taxonomy to discuss conceptual and architectural characteristics of each of the classes.
3 The Conceptual Point of View In this section, we discuss the conceptual point of view of our framework. To do so, we will take the specification language perspective, which we explain below. Then, the various classes of our taxonomy are discussed from this language perspective. In the conceptual point of view, we are interested in the conceptual specification of transactional workflows formulated in one or more specification languages. Given the two main classes in the taxonomy of Figure 1, we can have two situations. In the first situation, there is a separate language for specifying workflow aspects, the workflow definition language (WFDL), and a separate language for specifying transaction aspects, the transaction definition language (TRDL). In the second situation, there is an integrated language for specifying both workflow and transaction aspects, the transactional workflow definition language (TRWFDL). If we have two languages, the languages can have two relations: either one language is a refinement of the other, or the two languages are orthogonal with respect to each other. If the two languages have a refinement relation, we have the following. A language offers primitives to specify transitions in a state space. A language L2 is a refinement of a language L1 if there is a notion of correspondence (a relation in the mathematical sense) between its state space and that of L1, and between its primitives and those of L1, such that the transitions specified by the primitives maintain the correspondence between states (see [11] for a further explanation). If the TRDL is a refinement of the WFDL, the WFDL level contains workflow attributes and the intermediate states at the TRDL level are related to transaction states. If the WFDL is a refinement of the TRDL, the TRDL level contains transactional attributes and the intermediate states at the WFDL level are related to control flow states. In the integrated approach, all aspects are merged into a single language, covering a state space that is the cross product of the two state spaces discussed above. 3.1
Separate Languages
The main reason for using two separate languages is separation of concerns in dealing with control flow and transaction aspects in complex applications. Below, we discuss the three basic classes of the separate models approach. In the WF/TR case, the control flow aspect is leading in the specification of transactional workflows. Low-level workflow semantics are based on transactional semantics of individual workflow tasks or groups of workflow tasks. Hence, the TRDL is a refinement of the WFDL. Primitives of the WFDL are mapped to primitives of the TRDL. Transaction semantics are often imported from the data management level – the TRDL is a sublanguage of a data manipulation language (DML) in this case. The WF/TR approach is taken in most commercial workflow management systems that
Transactional Workflows or Workflow Transactions?
63
TASK task1 BUSINESS TRANSACTION USES FORM form1 END TASK
TRDL
WFDL
support (usually limited) transactional behavior of workflows. Below, we show a simple example in which individual workflow tasks can be parameterized to behave as business transactions (atomic and isolated units of execution). On the left, we see the specification of a workflow task. The second and third lines of this specification are expanded on a lower abstraction level to the transaction specification shown on the right. When executed, the TRDL specification will induce intermediate states with respect to the WFDL specification. BEGIN TRANSACTION READ form1.field1 READ form1.field2 USE form1 WRITE form1.field1 WRITE form1.field2 IF status_ok THEN COMMIT TRANSACTION ELSE ABORT TRANSACTION END TRANSACTION
TRANSACTION tr1 EXECUTE ATOMIC IMPLEMENTATION wf1 END TRANSACTION
WFDL
TRDL
In the TR/WF class, transactional behavior is the leading aspect in the specification of transactional workflows. High-level transactional semantics are specified with a workflow as elaboration of the underlying process structure. Hence, the WFDL is a refinement of the TRDL. The TR/WF approach is applied for example in workflow management for e-commerce applications. Here, the transaction between two business partners is the starting point and the elaboration of the control flow a refinement of the transaction. We show a simplified example below. On the left, we see a TRDL specification of a transaction that states transactional properties. The control flow is seen as an implementation detail to be specified at a lower level of abstraction. This is elaborated in the WFDL specification on the right. Note that the WFDL specification concerns a non-linear process, which is not easy to specify in traditional TRDLs. The execution of the WFDL specification will introduce intermediate states with respect to the execution of the TRDL specification. WORKFLOW wf1 TASK task1 task2 task3 task4 SEQUENCE task1 task2 SEQUENCE task1 task3 SEQUENCE task2 task4 SEQUENCE task3 task4 END WORKFLOW
In the TR+WF approach, there is a balance between control flow and transactional behavior. High-level transactional semantics are defined on the same conceptual level as workflow processes. Hence, workflow and transaction specifications refer to each other on the same level of abstraction. Below, we show a stylized example. On the left, we see a WFDL specification that specifies a control flow and refers to the TRDL specification for the transactional properties. The TRDL specification shown on the right imports the task list from the WFDL specification and specifies transactional properties over this. The TRDL specification specifies compensating tasks [5] and a safepoint [10] to allow flexible rollback by compensation. Control flow and compensation functionality can be changed independently of each other, thus creating a separation of concerns between workflow and transaction specification.
3.2
P. Grefen WORKFLOW wf1 REFERS TRANSACTION tr1 TASK task1 task2 task3 SEQUENCE task1 task2 SEQUENCE task2 task3 END WORKFLOW
TRDL
WFDL
64
BEGIN TRANSACTION tr1 REFERS WORKFLOW wf1 COMP ctask1 task1 COMP ctask2 task2 SAFEPOINT task1 END TRANSACTION
Integrated Models
TRWFDL
In the integrated model class of the taxonomy, workflow and transaction semantics are combined into one single model. In the TRWF class of our taxonomy, we find hybrid workflow and transaction models. These models are reflected in hybrid transactional workflow specification languages. These languages contain typical workflow-related primitives – e.g., to express control flows – and transaction-related primitives – e.g., to express atomicity or isolation requirements. An obvious way to create a TRWF language is to ‘merge’ a pair or languages of the TR+WF class. Following this approach, we can obtain the example below from the TR+WF example shown above. Clearly, the TRWF and TR+WF approaches are exchangeable to some extent. WORKFLOW wf1 TASK task1 COMP ctask1 SAFEPOINT TASK task2 COMP ctask2 TASK Task3 COMP none SEQUENCE task1 task2 SEQUENCE task2 task3 END WORKFLOW
WFDL
In the WF class, transactional semantics are expressed in workflow processes. Specific process patterns are used to express transaction behavior of workflow processes. An example is the specification of compensation patterns in workflow definitions to achieve relaxed atomicity characteristics for a workflow. We show an example below. In the WF specification, we see the definition of regular tasks and a regular control flow (three consecutive tasks) and the definition of compensating tasks and compensating control flow (two consecutive tasks). The compensating control flow is linked to the regular control flow through or-splits (alternative paths). At an or-split, a condition is evaluated to check whether rollback of the workflow is required – if not, the regular control flow is followed – if so, the compensating control flow is followed. Note that the example is in fact a static specification of all possible cases of the dynamic compensation behavior of the TRWF example above. WORKFLOW wf1 TASK task1 task2 task3 TASK ctask1 ctask2 SPLIT or1 or2 SEQUENCE task1 or1 SEQUENCE or1 task2 SEQUENCE task2 or2 SEQUENCE or2 task3 SEQUENCE or1 ctask1 SEQUENCE or2 ctask2 SEQUENCE ctask2 ctask1 END WORKFLOW
# regular tasks # compensating tasks # start regular control flow
# start compensation control flow
In the TR class of the taxonomy, workflow semantics are expressed in transaction specifications. In this approach, transactions have structured processes as their action
Transactional Workflows or Workflow Transactions?
65
TRDL
specification. An example is shown below. Here we see a transaction consisting of two subtransactions that can be executed in parallel – thus constituting a rudimentary form of control flow. TRANSACTION tr1 SUBTRANSACTION s1 action1; action2 END SUBTRANSACTION SUBTRANSACTION s2 action3; action4 END SUBTRANSACTION PARALLEL s1 s2 END TRANSACTION
4 The System Point of View After having discussed the conceptual point of view in the previous section, we turn to the system point of view in this section. Where the conceptual point of view explains the ‘what’, the system point of view explains the ‘how’ – i.e., the support of workflow and transaction models. We base the system point of view on the architecture aspect, focusing on the highlevel structure of transactional workflow support systems. We use abstract architectures to identify the elementary system characteristics of the classes of the taxonomy. We relate these abstract architectures to concrete architectures in Section 5. In the description of the architectures, we place workflow management and transaction management modules on top of a function and data support (FDS) layer. The details of this layer are not relevant in the context of this paper. Below, we turn to the various classes of transactional workflows, again organized as depicted in Figure 1. 4.1
Separate Models
In the separate models category of our taxonomy, we have separate workflow and transaction management modules in the architecture (WFM respectively TRM). These modules can have three architectural relations (as depicted in Figure 2): WFM as client of a TRM server, TRM as client of a WFM server, and WFM and TRM as peerto-peer modules. These three architectures coincide with the three classes WFM/TRM, TRM/WFM and TRM+WFM. In discussing the characteristics of the three classes, the focus is on the WFM-TRM interface, as indicated by triangles in Figure 2. This interface is used in all three architectures to synchronize the control flow state in the WFM and transaction state in the TRM. The WFM/TRM architecture is depicted in the left hand side of Figure 2. The interface between WFM and TRM is both a control and a data channel. The WFM uses the TRM interface to open a transaction context and perform data manipulation operations in this context. In this class, TRM and FDS are often integrated into one database application environment based on a DBMS with built-in transaction management functionality. The WFM/TRM architecture is ‘standard’ for commercial systems.
66
P. Grefen
WFM WFM
TRM TRM
TRM TRM
WFM WFM
FDS FDS
FDS FDS
TRM TRM
WFM WFM
FDS FDS
Figure 2. WFM/TRM, TRM/WFM and TRM+WFM architectures
The TRM/WFM architecture is depicted in the center of Figure 2. The interface between TRM and WFM is both a control and a data channel. The TRM uses the WFM interface to open a workflow context and next to invoke control flow primitives. We observe this architecture class in e-commerce environments where a high-level transaction engine invokes processes supported by workflow management technology. The TRM+WFM architecture is depicted in the right hand side of Figure 2. In this architecture, we have a TRM as transaction server and a WFM as process server in a peer-to-peer relation. The interface between WFM and TRM is strictly a control channel: the WFM communicates process states to the TRM, the TRM communicates transaction contexts and workflow commands to effectuate transactional effects on process states to the WFM. Note that this interface is not a standard interface as defined by the WfMC – its Interface 4 standard describes communication between two workflow servers [17]. The TRM+WFM architecture is – trivially – fit for TR+WF language support. TRWF language support is possible by filtering a TRWF specification into the right parts for TRM and WFM. 4.2
Integrated Models
In the integrated models class of our taxonomy, we have a single transactional workflow management module in the architecture. This can either be a transactional workflow manager (TRWFM), a traditional workflow manager (WFM), or an advanced transaction manager (TRM). The three cases are shown in Figure 3. The TRWFM offers integrated support for transaction management and workflow management. A hybrid transaction and workflow state is maintained within the TRWFM. It supports languages in the TRWF and TR+WF classes. To handle TR+WF specifications, the two subspecifications are merged into one specification by a preprocessor. In the WFM architecture class, the state of a transactional workflow is completely maintained by WFM. Transactional attributes of this state are mapped to workflow attributes. The WFM can only interpret specifications in the WF class. Specifications in other classes (typically TRWF) have to be translated to WF format. This architecture class is fit for support by commercial workflow management systems. In the TRM class, the state of a transactional workflow is completely maintained by the TRM. Control flow attributes of this state are implemented by transaction attributes. The TRM can only interpret specifications in the TR class. Specifications in other classes (typically TRWF) have to be translated to TR format. Nested process
Transactional Workflows or Workflow Transactions?
TRWFM TRWFM
WFM WFM
TRM TRM
FDS FDS
FDS FDS
FDS FDS
67
Figure 3. TRWFM, WFM and TRM architectures
structures can be supported by standard transaction management technology. More advanced structures are typically only supported by research prototypes.
5 Application of the Framework In this section, we apply the taxonomy we have developed in two ways: we present a comparison of the various classes with respect to their characteristics and we place existing work in our taxonomy. 5.1
Comparing the Classes
In Table 1, we show a comparison of the classes in our taxonomy. For each class, we list the main goal, the means used to achieve this goal, and a brief list of advantages and disadvantages – which we explain in the sequel. Flexibility in coupling models and separation of concerns between workflow and transaction aspects are main advantages of the classes with separate models. Problems with integration of models (and support based on these models) form the downside of this aspect. Consistency of specifications is a main advantage of the single-model approach: there are no separate models to be kept consistent. Consistency can certainly be a problem in the TR+WF class, as two specifications of a transactional workflow exist without a ‘leading’ specification. Limited expressiveness is a clear disadvantage of the WF and TR classes, as possible semantics of transaction aspects are limited by available workflow primitives and vice versa. The TRWF class does generally not have this problem, but a complex formalism with equally complex semantics usually is the basis for models in this class. Finally, system support is (curTable 1. Comparison of classes class
goal WF with robust character TR with complex TR/WF control flow integrated WF TR+WF and TR integrated WF TRWF and TR WF with robust WF character TR with complex TR control flow WF/TR
means data management in WFs process management in TRs coupled process and data mngmnt hybrid process and data management advanced process management advanced TR management
advantages sep. of concerns, flexibility, support separation of concerns, flexibility separation of concerns, flexibility integration, consistency simple formalism, consist., support simple formalism, consistency
disadvantages integration integration integration, consistency complex formalism, inflexibility limited expressiveness limited expressiveness
68
P. Grefen
rently) best for the WF/TR class (supported by combinations of existing WFM and TRM systems) and the WF class (implementable on commercial WFM systems). 5.2
Positioning Existing Approaches
In this subsection, we place a selection of existing approaches in our taxonomy. For reasons of brevity, the overview is far from complete. We first discuss approaches in the separate models category, then turn to the integrated model category, and finally pay attention to an approach that combines aspects of both categories. A more elaborate analysis is presented in [11]. In the Mercurius initiative [8], a WF/TR architecture has been proposed: workflow management functionality is placed on top of transaction management functionality. The CrossFlow approach [9, 16] is an example of the TR/WF class. In the CrossFlow language perspective, cross-organizational transactions are specified in an electronic contract that is mapped to workflow definitions. In the CrossFlow architecture perspective, inter-organizational transaction management functionality is placed on top of workflow management systems. In [4], an approach to a language in TR+WF class is proposed in which independence between specification of control flow on the one hand and transactional characteristics on the other hand is a starting point. In the context of the FlowMark WFMS, an approach for the support of business transactions has been developed [14]. It uses specification of transactional spheres in a workflow process that is interpreted by extended workflow technology – the approach can hence be placed in the TRWF class. ObjectFlow [12] can also be placed in the TRWF class. In the Exotica approach [1], a language is proposed in TRWF class. This language is preprocessed to be interpretable by an architecture in the WF class. In the FlowBack project [13], a similar approach has been developed. In the WF language class, we find work in which transactional characteristics are coded into Petri Nets [3]. The ConTracts model [15] is an approach in the TR class providing an environment for reliable execution of long-lived computations. The control flow primitives of the ConTracts language have been used for the realization of transactional workflows. TSME [6] is an approach in the TR class with comparable goals. In the WIDE project, an approach has been developed that combines aspects of both main classes of our taxonomy. The WIDE workflow specification language belongs in the TRWF class, as it is a ‘classical’ workflow definition language extended with (among other things) transactional primitives. The WIDE architecture contains three levels [7]. The two high-level transaction management levels GTS [10] and LTS [2] belong in the TR+WF class, the low-level LTI level [2] belongs in the WF/TR class.
6 Conclusions and Outlook In this paper, we have described a taxonomy for transactional workflow support, paying attention to both the conceptual and system point of view. Characteristics of the conceptual point of view have been described in terms of specification language classes. System characteristics have been discussed in terms of architecture topologies and interfaces. The result is a taxonomy that provides a background for selecting or analyzing transactional workflow support. Choosing appropriate classes from our taxonomy for the conceptual and system points of view is a basis for the configuration
Transactional Workflows or Workflow Transactions?
69
of transactional workflow support in complex applications, where functional requirements and architectural context both play an important role. The choice can be different in both points of view to some extent, but a mapping must be possible. Completing the analysis of existing approaches to obtain an overview of the state of the art is an obvious follow-up of the presented work. Extending the framework to better cover multi-level transaction support is an important research direction. Acknowledgments. Maarten Fokkinga is acknowledged for his assistance with respect to language concepts and his feedback on the draft version of this paper.
References 1. G. Alonso et al.; Advanced Transaction Models in Workflow Contexts; Procs. Int. Conf. on Data Engineering, 1996; pp. 574-581. 2. E. Boertjes et al.; An Architecture for Nested Transaction Support on Standard Database Systems; Procs. 9th Int. Conf. on Database and Expert System Appls., 1998; pp. 448-459. 3. J. Dehnert; Four Systematic Steps towards Sound Business Process Models; Procs. 2nd Int. Colloq. on Petri Net Techn. for Modeling Comm. Based Systems, 2001; pp. 55-63. 4. W. Derks et al.; Customized Atomicity Specification for Transactional Workflows; Procs. 3rd Int. Symp. on Cooperative Database Systems for Adv. Appls., 2001; pp.155-164. 5. H. Garcia-Molina, K. Salem; Sagas; Procs. 1987 ACM SIGMOD Int. Conf. on Management of Data, 1987; pp. 249-259. 6. D. Georgakopoulos, M. Hornick, F. Manola; Customizing Transaction Models and Mechanisms in a Programmable Environment Supporting Reliable Workflow Automation; IEEE Trans. on Knowledge and Data Engineering, (8)4, 1996; pp. 630-649. 7. P. Grefen et al.; Two-Layer Transaction Management for Workflow Management Applications; Procs. 8th Int. Conf. on Database and Expert System Appls., 1997; pp. 430-439. 8. P. Grefen, R. Remmerts de Vries; A Reference Architecture for Workflow Management Systems; Journ. of Data & Knowledge Engineering, (27)1, 1998; pp. 31-57. 9. P. Grefen et al.; CrossFlow: Cross-Organizational Workflow Management in Dynamic Virtual Enterprises; Int. Journ. of Computer Systems Science & Engineering, (15)5, 2000; pp. 277-290. 10. P. Grefen, J. Vonk, P. Apers; Global Transaction Support for Workflow Management Systems: from Formal Spec.to Practical Impl.; VLDB Journal, (10)4, 2001; pp. 316-333. 11. P. Grefen; A Taxonomy for Transactional Workflows; CTIT Technical Report 02-11; University of Twente, 2002. 12. M. Hsu, C. Kleissner; ObjectFlow and Recovery in Workflow Systems; in: V. Kumar, M. Hsu; Recovery Mechanisms in Database Systems; Prentice Hall, 1998; pp. 505-527. 13. B. Kiepuszewski, R. Muhlberger, M. Orlowska; FlowBack: Providing Backward Recovery for WFMs; Procs. ACM SIGMOD Int. Conf. on Management of Data, 1998; pp. 555-557. 14. F. Leymann; Supporting Business Transactions via Partial Backward Recovery in WFMs; Procs. Datenbanksysteme in Büro, Technik und Wissenschaft, 1995; pp. 51-70. 15. A. Reuter, K. Schneider, F. Schwenkreis; Contracts Revisited; in: S. Jajodia, L. Kerschberg; Adv.Transaction Models and Architectures; Kluwer Academic, 1997; pp. 127-151. 16. J. Vonk et al.; Cross-Organizational Transaction Support for Virtual Enterprises; Procs. 5th Int. Conf. on Cooperative Information Systems, 2000; pp. 323-334. 17. Workflow Management Coalition Workflow Standard – Interoperability Abstract Specification; Doc. No. WfMC TC-1012; Workflow Management Coalition, 1996.
Collaboration Flow Management: A New Paradigm for Virtual Team Support Jacques Lonchamp LORIA, BP 239, 54506 Vandœuvre lès Nancy Cedex, France [email protected]
Abstract. Collaboration flow management is a new paradigm for virtual team support which departs from the classical ‘workflow management’ and ‘collaborative computing’ paradigms. The aim is to support the opportunistic flow of collaboration within a distributed project, considered as a living and selforganizing system. Such a flow includes informal and formal, synchronous and asynchronous, task-oriented and project-oriented collaborative sessions. Some of them are elements of model-driven session-based process fragments. The paper defines the paradigm and describes our Java prototype of collaboration flow management system through a realistic scenario.
1 Introduction A virtual team (VT) is a team distributed across space, time, and organization boundaries, and linked by webs of interactive technology. As every team, a VT is a group of people who work interdependently with a shared goal. Teams exist for some task-oriented purpose, and differ from social groups (virtual communities, electronic groups) and decision groups (virtual government, direct participation). The growing importance of the VT concept is related to several well known reasons: the diversification of organizational structures towards more distributed, decentralized, and flexible organizations, the globalization of the economy, with the development of worldwide organizations and projects, the rapid growth of telecommunications and Internet in particular. Cross-organizational VTs, such as virtual enterprises, or networked projects, are the most in need of new ways to work and technology-support infrastructures. VTs can improve work performance by reducing costs (cutting travel costs and time), shortening cycle time (moving from serial to parallel processes), increasing innovation (permitting more diverse participation, and stimulating creativity), and leveraging learning (capturing knowledge in the natural course of doing the work, gaining wider access to expertise, and sharing best practices). Communication tools, especially those aiming at geographical distribution transparency, such as video/audio conferencing systems, are obviously useful for VTs but not sufficient. The success of a VT depends on a set of psychological, organizational, and cognitive factors [11]. A computerized support for VTs should take into account these factors and include them in its basic requirements. First, a sense of identity and R. Cicchetti et al. (Eds.): DEXA 2002, LNCS 2453, pp. 70–80, 2002. Springer-Verlag Berlin Heidelberg 2002
Collaboration Flow Management: A New Paradigm for Virtual Team Support
71
membership is necessary for VTs. Periodically performing informal virtual meetings is not sufficient for developing a collaborative attitude. Participants must feel that the supporting environment helps them to do important and constructive things together, in synergy (R1). Secondly, traditional authority is minimized in VTs, which develop an inner authority based on competencies. In VTs, power comes from information, expertise, and knowledge, not from position. The important things done together through the supporting environment should include expression of competencies, externalization of knowledge and mental models related to the task in hand (R2). Third, trust is the key to VTs. People work together because they trust one another. People make deals, set goals, and lend one another resources. Without face-to-face cues, and the nuances of person-to-person interaction, trust is harder to attain. But trust can be built with the recognition of the contribution that everyone makes, the clarity of positions and commitments (R3). Finally, project and process management is a critical ingredient for successful distributed work. Co-located teams can quickly clarify goals, correct misunderstandings, and work through problems. VTs need to be more explicit in their planning and their processes (R4). When considering these requirements, we claim that no existing paradigm for cooperative systems, neither the ‘workflow management paradigm’ nor the ‘workgroup computing paradigm’, is satisfying. A new paradigm is required for VT support. The second section of this paper elaborates this idea and defines such a paradigm, called the collaboration flow management paradigm. The third section describes our current Java prototype of collaboration flow management system through a realistic scenario. Finally, the last section compare our proposal with related work and draw some conclusions.
2 The Collaboration Flow Management Paradigm The fourth requirement (R4) stresses the importance of supporting the VT life cycle process. VT projects usually follow a life cycle with an alternation of divergence phases, during which people work individually, and collaborative convergence phases, during which people build some shared understanding, discuss for discovering and solving the divergences accumulated during individual work, and drive the project. Cognitive ergonomists draw a similar distinction for collective design tasks between individual design steps, requiring operative synchronization mechanisms, and collaborative co-design steps, requiring cognitive synchronization mechanisms [4]. A classical Workflow Management System (WfMS) provides support for coordinating individual activities, as those occurring during divergence phases. Generally, a WfMS provides no support for collaborative activities, as those constituting convergence phases. The three first requirements (R1 to R3) imply that a VT environment should mainly support convergence phases. The ‘workgroup computing paradigm’ is not a satisfying solution because it considers collaborative tasks in isolation and does not support the life cycle process as a whole. A computerized environment for VTs should support more or less structured processes whose steps are collaborative sessions either synchronous or asynchronous, informal or formal. On the basis of re-
72
J. Lonchamp
quirements (R1 to R3), we analyze these sessions as working sessions, during which participants express their views, competencies, positions, and commitments about the task in hand. Cognitive ergonomists distinguish between ‘production activities’, for the collaborative production of artifacts (idea lists, concept graphs, ontologies, design sketches,…), and ‘evaluation activities’, for the collaborative evaluation and integration of individually produced artifacts. We consider also collaborative sessions for project orientation and project organization (objectives, to do lists, plans,…). At a very abstract level, we feel that issue resolution is the basic building block which can be used for describing all these activities. Participants discuss and solve various issues about the artifacts, the shared context, and the collaboration process itself. The computerized environment should support the definition and management of more or less structured processes whose steps are issue-based synchronous or asynchronous, informal or formal, collaborative sessions. Otherwise, a VT is a living and self-organizing system. For Peter and Trudy Johnson-Lenz [7], post-mechanistic groupware, like all living systems, are ‘rhythmic’ and made of ‘containers’ with flexible and permeable ‘boundaries’. The VT computerized environment should support in particular a range of session (containers) types providing different collaboration rhythms, in accordance with the task urgency and the required depth of issue analysis, and different group boundaries, through the evolving definition of a ‘core team’ and an ‘extended team’. Core team members are strongly involved in the project and use directly the computerized environment. Extended team members should only be involved through standard communication technologies, such as annotating Web pages or participating to Usenet-like forums. Procedures (process models), context, and timing are the three other primitives of postmechanistic groupware [7]. The choice between these forms (containers, rhythms, boundaries, procedures) should be made dynamically and collectively during the course of the project. From above, the following four basic functionalities can be identified. (F1) Support different types of short/rapid, more or less formal, synchronous sessions for the core team, such as artifact-centered informal sessions (around a shared document or a free hand drawing) or formal argumentation sessions (with different rhythms and styles of interaction: free, turn talking, circle, moderated). Composite synchronous sessions can mix formal and informal times. (F2) Support longer/slower/deeper, formal, asynchronous sessions for the core team, controlled through the enactment of issue-based and session-based collaborative process model fragments. Such model fragments can specify for instance different structured brainstorming processes, document inspection processes, knowledge elicitation processes, solution evaluation processes (comparative, analytical, analogical). When needed, an asynchronous issuebased session within a process fragment can be transformed into a synchronous session for continuing with a faster rhythm the resolution of conflicting issues. (F3) Support the overall collaboration flow: some asynchronous Project Management (PM) process enables project initiators to launch synchronous sessions and asynchronous process fragments (see Fig.1). This PM process is the spring of the opportunistic flow of collaboration that the VT generates, which can include in particular other PM processes better suited to the circumstances and which replace the initial one (see
Collaboration Flow Management: A New Paradigm for Virtual Team Support
73
section 3.3). (F4) Play the role of a project memory, storing all project related information: artifacts, debates, free hand drawings, messages, process trace, etc. These elements should be externalized as HTML/XML documents, for easier access and feedback from the extended team during asynchronous informal sessions on the Web (see Fig.2). Synchronous session
launch
Initial asynchronous PM process launch
replace
Changed PM process
Asynchronous sessions
Asynchronous session-based process fragment
replace transform
Fig. 1. Main elements of a collaboration flow Session types synchronous informal synchronous formal asynchronous formal asynchronous informal
Typical examples artifact-centered informal session formal argumentation session instance of a session type within an issue-based and session-based collaborative process model fragment informal annotation of ‘externalized’ artifacts using the Web
Fig. 2. Summary table of the four basic session types with typical examples
3 A Prototype of Collaboration Flow Management System In this section, we describe a scenario supported by our current prototype of Collaboration flow Management System (CfMS). Our lab plans an extension of its building. A VT has been created for discussing different aspects of this project with the point of view of the future residents. The team includes representatives of lab members and participants coming from the university, the town, and the firm of architects. A kickoff informal ‘orientation meeting’ has first planned a set of virtual meetings about several specific questions. These synchronous sessions are organized by the project manager through the asynchronous PM process (see section 3.3).
3.1. Synchronous Session Support The session described here is about car parking near the lab. Such a synchronous session does not follow any predefined execution path: the session moderator can describe the problem and organize the work by using the audio channel. In our scenario, the problem is first informally discussed through textual or graphical annota-
74
J. Lonchamp
tions on the overall building plan (see Fig.3). Social protocols, such as author identification with colors, can be negotiated. Written clarifications may be obtained through the chat tool. In a second time, emerging controversial issues can be formally discussed with pro and con arguments, for instance by using a ‘turn talking’ policy (‘free’ and ‘moderated’ policies are also available). On the basis of the resulting argumentation trees (described in more details in section 3.2.2), the session moderator can select some solutions and measure the consensus within the core team with the voting tool. All produced documents (drawings, annotated documents, argumentation trees, vote results) can be saved as HTML documents for feedback from other lab members.
(4)
(1) (2)
(3)
Fig. 3. The synchronous session support, with a formal argumentation (1), a vote result (2), an informal discussion (3) about the freely annotated architectural plan of the lab (4).
3.2. Model-Driven Collaborative Process Fragment Support Later in the course of the project, the VT has planned an asynchronous model-driven process fragment for a systematic study of the existing cafeteria dysfunctions and for proposing a consensual list of possible causes and solutions for the new cafeteria. 3.2.1. The Meta Model The process modeling language is defined by a meta model, which extends the classical process view and organizational view of WfMSs with a decision view and a knowledge view (see Fig.4). The process view mainly includes ProcessFragment and Session types. Each process fragment instance is structured into a network of session instances (also called ‘phases’), with precedence relationships, and special phase instances corresponding to the classical workflow operators (AND split, OR split, AND join, OR join phase types). The knowledge view mainly includes Artifact, Component, and ApplicationTool types. Artifact types specialize generic types such as Text, List, Table, Graph (concept map), or Image. Component types specialize ge-
Collaboration Flow Management: A New Paradigm for Virtual Team Support
75
neric types such as ListElement, GraphNode, or GraphVertex. ApplicationTool types mirror the specialization of artifact types, with specializations of generic types such as TextViewer, ListViewer, TableViewer, or GraphViewer. Each Session type grants access to some ApplicationTool types. The organizational view mainly includes Role and Actor types. The decision view describes collaborative work at a fine grain level. Each Session type is defined internally by a set of Issue types, which must be solved either individually or collectively. Each Issue type is characterized by a set of Option types, which describe the possible solutions. At run time, one option is chosen trough an argumentation process. Each Option type can trigger, when it is chosen, some Operation type, which can change some Artifact or Component, or change the process state (e.g. termination of a Session), or change the process definition itself. SpecializaAggregation Relationship (class level) Relationship (instance level)
ProcessFragment Type
Terminates
Changes
precedes
Contains Has Session Issue Option Type Type Type
Triggers
Operation Type
for/against Argument Type
AND split
for/against
AND join OR split OR join
Constraint Issue
from Constraint Type to
Artifact Type
Component Type Node
Text Graph List
Link
Too Type
Role Type
Text Viewer
Actor
List Viewer
Element Graph Viewer
Can Raise, Can Participate, Can Solve GiveAccessTo
Can Launch
Fig. 4. The basic concepts of the meta model
3.2.2. The Scenario A process model fragment is selected and instantiated (see section 3.3). The fragment implements the DIPA approach [10] which formalizes collaborative problem solving for synthesis situations (design of solutions), and for analysis situations (diagnosis). A typical DIPA-based analysis process includes three steps: first, the problem is described and data (‘symptoms’) are collected; secondly, data are used to find interpretations, i.e. possible causes; lastly, interpretations are used to devise solutions that serve as reparations suppressing the causes of the symptoms. The process model fragment includes three session types. The ProblemDefinition session type contains issues types for defining the problem (individual), proposing a symptom (individual), and terminating the phase (collective). The ProblemAnalysis session type contains issue types for proposing a cause (individual), linking a cause to a symptom (individual), challenging -keeping or discarding- a cause (collective), adding new symptoms
76
J. Lonchamp
(individual), and terminating the phase (collective). The ProblemResolution session type contains issue types for proposing a solution (individual), linking a solution to a cause (individual), challenging a solution (collective), and terminating the phase (collective). The instantiated process is a sequence of one phase of each type. When a user, say Jack, logs in a process fragment where he plays a role, he can use the What can I do? guidance query and the Whats’new? awareness query (for knowing elements that have changed since his last logout). He can also browse various textual and graphical representations of the model types and the process instances. All participants only act by solving typed issue instances. If Jack wants to propose a new cause, he raises an individual ProposeCauseIssue. Then he has to explain his intention as an argument. This individual issue is automatically solved and triggers an operation which creates a new Cause component. If Jack wants to challenge a cause given by Peter, he raises a collective ChallengeCauseIssue. Then, he gives an argument in favor of the Discard option. Other participants can react (support the Keep option or challenge Jack’s argument). The debate takes the form of an argumentation tree, possibly with qualitative importance constraints between arguments. Such a constraint (‘more/ less/equally important than’) opens a sub issue, called a ‘constraint issue’, for debating about the value of the constraint. Participants can access various representations of argumentations, like a threaded discussion form (Fig.3) or a graphical form with a ‘playback’ facility (Fig.5). At each argumentation move, the system computes a preferred option by using an argumentative reasoning technique close to the approach proposed in Zeno [5]. The system provides also statistical charts about the participation, the favorite option of each participant, and the ‘level of conflict’, based on the number of changes of the preferred option. Suzan, who plays a role giving the capability to solve this type of issue, terminates the debate by choosing one option. The issue resolution triggers an operation that either keeps or discards the challenged cause. Different tools, like table or graph viewers, are available to the session participants for accessing the process artifacts, such as the symptom/cause/solution concept graph of Fig.5. The process guidance, the argumentation and decision-making support, the awareness facilities, and the way to transform an asynchronous debate into a synchronous one are described in more details in [12], which also gives some information about the current Java implementation. A model fragment can include ‘open points’ for refining the model at run time. In our model an open point is an instance of a model refinement phase type including a model refinement issue type, with one option type for each possible refinement. At run time, one of these options is chosen, either individually by an actor playing the role of project manager, or collectively by the core team. The resulting operation deploys the refinement by instantiating new phase instances and new phase precedence relationships. By this way, artifact production and process refinement are performed similarly. 3.3 The Project Management Process In our scenario, the initial PM process follows a very simple model with a PM phase followed by a refinement phase. During the PM phase a participant playing the role of
Collaboration Flow Management: A New Paradigm for Virtual Team Support
77
ProjectManager can plan synchronous sessions and asynchronous process fragments by solving CreateSessionOrProcess individual issue instances. Each resolution adds a new description (name, date, objective, participants, status,…) to a ProjectPlan artifact and calls people for participation through email generation. This planning activity can also result from synchronous ‘orientation meetings’ with the core team. The ProjectManager can also publish in a ProjectRepository artifact, HTML documents produced by the terminated processes/sessions (PublishResult issue type). A dedicated tool for accessing each document is generated and appears in the tool list of the PM session (see Fig.6). The refinement phase enables the project manager (possibly after a debate with the core team) to change the PM policy by instantiating a new PM fragment. For instance, a richer model can include an issue type for publicizing individual tasks by publishing them in the ProjectPlan artifact. Participants can commit themselves to do a given task, and their commitments are published in the ProjectPlan. Model-driven process fragments can also be used within a PM session for defining and installing dynamically (‘on line’) and consensually new model fragments as it is proposed in [6]. It is perhaps not a realistic approach because participants should be aware of all the details of the modeling approach and language. Another way is to use informal sessions for debating the general characteristics of the process model fragment, possibly around high level graphical descriptions. The model programming and installation is then performed ‘off line’ by model developers using the specialized model development environment (editor, compiler, static analyzer, and instantiation tool [12]).
(3)
(2) (4)
(1)
(6) (5)
Fig. 5. The asynchronous process-centered session support, with a graphical view of an issue (1), the DISA concept graph built during the session (2), the action panel (3) with a ‘What Can I do?’ guidance query (4), the notification panel (5), and the statistical analysis of the debate.
78
J. Lonchamp
(3) (1)
(2) Fig. 6. The asynchronous project session support with the project plan (1) and the project repository (2). A dedicated tool is generated for each published result (3).
4 Conclusions and Future Work Our approach is rooted in a wealth of research projects. Among the most important precursors, we can cite Conversation Builder [8] (for genericity, process model fragments, asynchronous collaboration), Cognoter/Argnoter [17] and gIBIS/rIBIS [3] (for argumentation and concept mapping). Our approach reflects also the evolution, in many application domains, from hard-coded tools to model-based generic environments. It is the case for instance for review/inspection systems, moving from specialized tools such as ICICLE, Scrutiny, etc. (see [15]) to generic environments such as CSRSv3 [18] or ASSIST [16]. Two recent systems have strong similarities with our proposal. First, we share with Compendium [2] the idea of modeling collaborative work through a set of typed issues, called ‘question-based templates’ in Compendium. We share with SCOPE [6] the idea of ‘session based process modeling’. SCOPE roughly coincides with our definition of a CfMS. But SCOPE does not provide high level interaction services such as our argumentation service. On the opposite, SCOPE aims at satisfying a larger set of system requirements. For instance, SCOPE provides a specific concurrency control mechanism for enabling sessions to access artifacts simultaneously and for preventing inconsistent changes. In our view, a CfMS only provides the core services for collaboration, such as the management of collaboration-oriented artifacts local to the different sessions. A document sharing system for distributed teams such as BSCW [1] or our Java Motu prototype (available at http://motu.sourceforge.net) should complement the CfMS for managing (sharing, versioning) application artifacts, such as the architectural plans in our scenario. By this way, collaboration is not tied to some particular document management policy.
Collaboration Flow Management: A New Paradigm for Virtual Team Support
79
Similarly, no support is provided for coordinating individual tasks. A cooperative WfMS could complement our CfMS for this aspect. The ‘collaboration flow management paradigm’ aims at satisfying the basic requirements of VT support, in particular fluidity. As many researchers we have first focused on technical solutions such as structural reflection and meta object protocols (in a early Smalltalk prototype [14]). In the current Java prototype, structural reflection (‘linguistic reflection’ [9]) is not a very practical solution in complex cases which require dynamic coding in the process modeling language followed by a long transformation process (starting with our specific compiler producing Java code followed by the persistency pre processor, the regular Java compiler, and the instantiation process). More realistically, PM process fragments include predefined reflective issue types for simple cases such as modifying the class level relationships of Fig.4 (e.g. adding/suppressing existing issue types or tool types accessibility from a given phase type) or creating new types when the code can be generated from a set of userprovided parameters (e.g. creating new role types). Today, we think that fluidity is a larger problem, with multiform technical and organizational aspects such as fluidity of group boundaries (core team vs extended team), fluidity of roles (in the context of each session), fluidity of containers (sessions) definition and organization (capability to start a formal debate asynchronously and to terminate it synchronously), fluidity in rhythms (between different styles of synchronous interaction), fluidity in formalization (from informal debates to formal argumentation, from to do list to model-driven processes), etc. Our approach deals with a large range of such technical and organizational aspects. Concerning the core services of CfMSs, future work will be directed towards a better support for the VT project memory by using XML technologies. Such a memory should store and restore both in human-readable and machine-readable forms all kinds of information, such as the collaboration artifacts, the process history, the argumentation trees, and even the process model fragments themselves. Another important objective is to provide a larger library of reusable process model fragments, either task-oriented or PM-oriented, and to evaluate them through scenarios and experiments, such as those already performed with our previous prototype [13].
References 1. Bentley, R., Appelt, W., Busbash, U., Hinrichs, E., Kerr, D., Sikkel, K., Trevor, J., and Woetzel, G.: Basic support for cooperative work on the World Wide Web. International Journal of Human-Computer Studies, 46 (1997) 827-846 2. Conklin, J., Selvin, A., Buckingham Shum, S., Sierhuis M.: Facilitated Hypertext for Collective Sense making: 15 Years on from gIBIS. Tech. Report KMI-TR-112, KMI (2001) 3. Conklin, J., Begeman, M.: gIBIS: A hypertext tool for exploratory policy discussion. ACM Transaction on Office Information System, 6, 4 (1988) 303-331 4. Falzon, P., Darses, F., Béguin, P.: Collective Design Processes. Proc. COOP’96, Juan Les Pins, France, INRIA (1996) 141-149 5. Gordon, T.F., Karacapidilis, N.: The Zeno argumentation framework. Proc. 6th Int. Conf. on AI and Law (ICAIL’97), Melbourne, Australia, ACM Press (1997) 10-18
80
J. Lonchamp
6. Haake, J.M., Wang, W: Flexible support for business processes: extending cooperative hypermedia with process support. Information and Software Technology, 41, 6 (1999) 7. Johnson-Lenz, P., Johnson-Lenz, T.: Post mechanistic groupware primitives: rhythms, boundaries and containers. Int. Journal Man-Machine Studies, 34 (1991) 395-417 8. Kaplan, S., Caroll, A.: Supporting collaborative processes with Conversation Builder. Computer communications, 15, 8 (1992) 9 Kirby, G.N.C., Morrison, R., Stemple, D.W.: Linguistic reflection in Java. Software Practice & Experience, 28, 10 (1998) 1045-1077 10. Lewkowicz, M., Zacklad, M.: MEMO-net, un collecticiel utilisant la méthode de résolution de problème DIPA. Proc. IC'99, Palaiseau, France (1999) 119-128 11. Lipnack, J., Stamps, J.: Virtual teams. John Wiley & Sons, New York (2000) 12. Lonchamp, J., Muller: Computer-Supported Deliberations for Distributed Teams. Proc. I2CS 2001, Ilmenau, Germany, LNCS, Springer-Verlag (2001) 167-174 13. Lonchamp, J., Denis, B.: Fine-grained process modeling for collaborative work support: experiences with CPCE. Journal of Decision Systems, 7 (1998) 263-282 14. Lonchamp, J.: Object-oriented collaborative process modeling. Technology of ObjectOriented Languages and Systems (TOOLS 16). Prentice Hall (1995) 17-27 15. Macdonald, F., Miller, J.: Automatic generic support for software inspection. Proc. 10th International Quality Week, San Francisco (1997) 16. Macdonald, F., Miller, J., Brooks, A., Roper, M., Wood, M.: A review of tool support for software inspection. Proc. 7th Int. Workshop on CASE (1995) 17. Stefik, M., Foster, G., Bobrow, D., Kahn, K., Lanning, S., Suchman, L.: Beyond the chalkboard: computer support for collaboration and problem solving in meetings. CACM, 1, (1987) 32-47 18. Tjahjono, D.: Building software review systems using CSRS. Tech. Report ICS-TR-95-06, University of Hawaii, Honolulu (1995)
Domain Knowledge-Based Automatic Workflow Generation Soon Ae Chun, Vijayalakshmi Atluri, and Nabil R. Adam CIMIC and MS/IS Department Rutgers University Newark, New Jersey, USA {soon,atluri,adam}@cimic.rutgers.edu
Abstract. The traditional workflow design paradigm relies heavily on humans who statically specify business processes. However, such a manual design approach is not suitable for many cases: (a) Inter-agency workflows that cross autonomous organizational boundaries require experts who possess knowledge required for defining workflows composed of services from the constituent organizations; (b) Customized workflows that require many variations make it infeasible and error-prone to predefine the complex workflow in advance. This paper presents an alternative design paradigm that allows automatic on-the-fly generation of a workflow schema. This dynamically integrated workflow composition approach utilizes conceptual ontologies of domain services (tasks) and domain integration knowledge that serves as a model for workflow integration (composition) rules. It also uses user profiles (preferences) as part of (transient) compositional rule base. We present a prototype system for “registering a new business” in the State of New Jersey that implements this automatic workflow design methodology.
1
Introduction
Workflow management systems (WFMS) have made it possible to automate the coordinated execution of different business processing activities. WFMS take over the responsibility of controlling the coordinated execution of tasks from human coordinators. This automation is possible with well-formed workflow specifications. Currently, tasks and interdependency conditions among tasks need to be fully specified by humans, before the workflow is executed. Various modeling techniques have been proposed [5,7], but the modeling process remains manual. Graphical tools [14] can be helpful, however, the graphical objects still need to be put together manually. The workflow design process typically requires to specify all the possible use cases and different execution environments. Once the workflow has been designed, modification of the workflow definition is costly and time consuming. Manual workflow specification needs automation, given the fact that workflow technology is widely being adopted for web-based B2B and
This work is partially supported by the National Science Foundation under grant EIA-9983468 and by MERI (the Meadowlands Environmental Research Institute).
R. Cicchetti et al. (Eds.): DEXA 2002, LNCS 2453, pp. 81–93, 2002. c Springer-Verlag Berlin Heidelberg 2002
82
S.A. Chun, V. Atluri, and N.R. Adam
B2C e-commerce transactions as well as eGovernment services in G2G, G2B and G2C. Often, government services span agency boundaries e.g., for new business registrations [1], welfare and social services [4], etc. The workflow for these services is referred to as inter-agency workflow. An inter-agency workflow is composed of component services provided by autonomous government agencies. The decentralized execution model proposed in [3] allows automatic execution with the collaborative coordination of participating agencies’ services and information systems, while maintaining autonomy of each agency. In this paper, we emphasize that inter-agency workflow design needs to be dynamic and automatic rather than static and manual. Our approach is based on a knowledge-based workflow design method. An ontology of services serves as a domain model for component services (tasks) and their relationships. An ontology of government regulations serves as a domain model for compositional rules. We present a dynamic workflow model that makes use of the tasks and rules in the ontology hierarchies, and of user profiles. Compositional rules consist of selection rules that select obligatory and preference tasks, and coordination rules that glue tasks together in order. Each rule is represented as Condition-Action pair. The composition algorithm evaluates compositional rules against a user profile, automatically generating the customized inter-agency workflow. This knowledge-based approach allows automating the complex inter-agency workflow design as well as provides a customized workflow that requires fewer evaluations at run time. Problem: In the following, we provide two example scenarios to illustrate the need for a dynamic and automatic workflow design method. Example 1. A developer, Bill, wishes to build a warehouse complex on his property on the bank of the Hackensack River in Little Ferry, NJ. His property of 15 acres (block 108.04, lot 2.04) is currently vacant and designated as “Light Industrial and Distribution B” zone. His property is only two feet below the maximum tide level. He will need to fill it with some material to secure a sound foundation. A workflow designer needs to know that (1) Bill needs to acquire a zoning certificate from the NJMC; (2) if the parcel falls into the so-called environmentally sensitive area, such as wetlands, riparian land, and flood plains, etc., Bill may need permits from local, state and federal agencies, including NJ Department of Environmental Protection (NJDEP) Waterfront Development Permit, Stream Encroachment Permit, Water Quality Certificate, Army Corps of Engineers Section 10 or Section 404 permit, Riparian Grant, etc. The designer needs knowledge on local, state, and federal regulations in order to properly design a workflow for Bill. In addition, the designer needs to consider not only Bill’s case, but many others with different geo-spatial characteristics. Specifying all the possible variations for every user in one workflow is not feasible. The next example illustrates the need for variant workflows for “busines registration process.”
Domain Knowledge-Based Automatic Workflow Generation
(a)
Input Application clerk
Certificate of Inc. Secretary of State
FEIN
Register for Tax
Employer ID
IRS
Div. of Revenue
Dept of Labor
Air quality permit
83
Employer Insurance Div. of unempolyment Disability Insurance
DEP General Permit DMV
(b)
Input Application clerk
FEIN
Register for Tax
IRS
Div. of Revenue
Fig. 1. Customized business registration process
Example 2. John Smith would like to start a new auto body repair shop in New Jersey. Jane Carlson would like to open a convenience store. John Smith wants to locate his business in downtown Newark, Essex county. Jane Carlson wants to have her shop near her home in Mercer county. John wants to have the business incorporated with several employees, while Jane wants to have it in sole proprietorship without any employees. According to these requirements, two different inter-agency service specifications are needed as shown in figures (1a) and (1b) for John and Jane, respectively. While both entrepreneurs need to file the tax-related business registration application at the Division of Taxation, their required steps and tasks differ. John will also need a certificate of incorporation and employee related insurance as well as an air quality permit. Our automatic workflow design methodology benefits the designer, and also citizens who without the automated design system need to find which regulations by which agency apply for their particular situations. This is time-consuming and complex process, as different regulations are enforced by different autonomous agencies. Moreover, there are few people with comprehensive knowledge of the regulations and services of different agencies. In Section 2, we present the domain knowledge-based workflow generation model Section 3 presents an algorithm to generate the workflow definition and describes our prototype system. Section 4 discuss related work, followed by conclusions and future research in Section 5.
2 2.1
Automatic Workflow Composition Model Task and Service Ontology
For service discovery and composition, services are described as service interfaces for external use and internal implementations. We argue that descriptions of individual services do not capture the inter-relationships among services. The linear set of individual services without any structures can result in inefficiency in service discovery and identification. For this reason, we use an ontology of services. An ontology is defined as a description of concepts and relationships that
84
S.A. Chun, V. Atluri, and N.R. Adam
Fig. 2. Component Hierarchy for a Service of opening a new business
exist among them [8]. For example, the knowledge of a composite service Open a New Business is represented as a component hierarchy (Figure 2). Each node represents a service, and each down link (up link) represents a has-component (component-of) relationship. The has-component relationship emanating at a service, s, defines all the component services of s. The knowledge of tasks and their relationships explicitly considered in our workflow generation model facilitates the search for appropriate component tasks via component-of relationships and provides necessary interface and connection information needed for composing integrated services. Definition 1. [Domain Services/Tasks Ontology] A domain service ontology SO is defined as a set of services S= {s1 , s2 , ...} where each si ∈ S is defined as a triple id, SA, SR, where id is a service identifier, SA is a set of service attributes and SR a set of relationships that are associated with si . The service attribute SA is represented as a 5-tuple si = a, P re, A, Input, Output, where a denotes the execution agent, or service provider of si , P re denotes the precondition set for si , A the set of activities (or operations) within si , Input the set of input parameters to si , and Output the set of output parameters of si . SR is a set of relationships {rel1 , rel2 , ...} where each reli is a triple name, N, mode with name for relationship name, N for a set of services that bear reli relationship with si and mode to denote whether relationship reli is obligatory or optional with tasks linked in the service ontology. We use the notation reli (si ) to retrieve all the services that bear the relationship reli to si , i.e. N . A relationship SR = {has-component component-of, implement } for si defines a set of all component services for si , a set of services that si is a component of, and a set of regulatory rules that si enforces, respectively.
Domain Knowledge-Based Automatic Workflow Generation
2.2
85
Ontology of Government Regulations
Government services implement federal, state and local regulations [12,9]. Obvious examples include: “a drivers license is required for vehicle operation,” “low income households are entitled to social benefits,” etc. However, some regulations are not so obvious. Moreover, as seen in examples in 1, regulatory knowledge required for designing an inter-agency workflow crosses the boundaries of local, state, and federal agencies. Regulatory rules may include different types: 1. Semantic Rules: These are the rules that affect the activities involved in a workflow. For example, an environmental protection regulation states that any business type, such as an autobody shop, that releases a certain level of spray paint into the air is required to obtain an air quality permit. 2. Spatial Rules: These rules are concerned with the geographic features of a business [10], e.g. (1) If a development project includes construction of a structure along, in, or across the stream channel, or 100 year flood plain of any stream, then a stream encroachment permit is required. (2) If a business is located in municipalities under a certain agency jurisdiction, then a development application needs to be filed and approved by that agency. 3. Temporal Rules: Temporal rules may state the absolute and relative deadlines for a task. For instance, if a public records filing for a new business entity is submitted, then tax registration forms must be submitted within 60 days. 4. Sequencing Rules: These rules specify the obligatory sequences among tasks. For example, obtaining a certificate of incorporation is required before business registration. The comprehensive knowledge about regulations is conceptually modeled as regulatory topic ontology (Figure 3). This topic-subtopic hierarchy allows the identi-
Fig. 3. Government Regulations Ontology
86
S.A. Chun, V. Atluri, and N.R. Adam
id
id
Rule
condition
related action
and or te 1
te n
reg
op s1
s2 ... sn
Fig. 4. Model of a compositional rule
fication of necessary information (e.g. type, location and kind of a new business) and regulation information. Each leaf node is associated with a regulatory rule. The rule nodes are modeled as Condition-Action relations ( Figure 4), where Condition is a logical expression and Action is an operation on services (e.g. insert, parallel, order). This rule node has a link to the text document containing the regulation. The rule node also has a link to actual services that implement it (i.e. an implemented-by relationship). For example, the leaf node on the subtopic of incorporated has a relationship is-subject-to pointing to a regulatory rule reg. The rule reg is associated with the text document containing regulations such as “If the business structure is incorporation, then it is required to obtain a certificate of incorporation.” It also has a relationship of implemented-by to a service “Obtain a certificate of incorporation” in the service hierarchy described above. The following definitions formalize the regulations ontology and composition rules. Definition 2. [Domain Rules Ontology] Domain rules ontology RO is defined as a set of rule topics O = {o1 , o2 , ...}. Each o ∈ O is defined as a 4 tuple < id, RA, Rel, R >, where id denotes an rule topic identifier, RA denotes a set of attributes, Rel a set of relationships that c bears, and R is a set of domain rules, {r1 , r2 , ...} where each ri ∈ R is defined as a triple of (id, body, reg) where id denotes the rule identifier and body is denoted as Condition-Action pair c, a where c and a are expressions for the antecedent and consequent of the rule, respectively. The regulation that is related to the rule is denoted by reg, which has a set of attributes {rat1 , rat2 , ...} and a set of rule relationships {rrel1 , rrel2 , ...}. The condition c is defined as a logical expression, called trigger expression, as follows: Definition 3. [Trigger Expression] Given an alphanumeric string variable v ∈ S, a function f ∈ F which is a set of predefined functions, a relational operation op ∈ OP ={<, , =, =, <, , ≥, ≤, ≥, ≤, ∈, ⊂, ⊆, ∈, ⊂, ⊆, ⊃, ⊇, ⊃, ⊇} and a literal l ∈ {S, N } for strings S and numbers N , the trigger expression te is defined as follows:
Domain Knowledge-Based Automatic Workflow Generation
87
1. (v op l) is a trigger expression 2. (f (x) op l) is a trigger expression 3. If te1 and te2 are trigger expressions, then ¬te1 , te1 ∨ te2 , and te1 ∧ te2 , are also trigger expressions. The function f represents predefined semantic, spatial and temporal functions such as amount(x), distance(x, y), after(x, y), etc. An example trigger expression is (number(employee) ≥ 1). The action a ∈ ri is an operation o ∈ {insert, order, parallelize, ...}, that can be applied to tasks T in a workflow. Following shows semantics of actions: insert(t1 ) is an action to insert task t1 ; order(t1 , t2 ) imposes an order: t1 before t2 ; parallelize(t1 , t2 ) allows t1 and t2 to be executed in parallel. The following is an example of a rule defined above: (R1, amount(spray-paint) ≥ 1/2 gallon per hour, insert(obtain air-qualitypermit) , REG1 ) R1 states that if the amount of spray paint used is more than half a gallon per hour, then insert task obtain air quality permit into the workflow and add the trigger expression amount(spray-paint) ≥ 1/2 gallon per hour as the precondition for the task. This rule is related to regulation REG1 . 2.3
User Profile Model
While the service ontology and regulatory ontology are static most of the time, the user profile varies from case to case. Our approach contains a user profile for customization as an important part of the model. The individual user profile is modeled as a goal and a set of preference attributes and their values. The user goal in the model identifies a domain (i.e., a set of services) for a composite service, and user preferences are used for composition rule evaluation to select component services. User preferences include functional preferences (e.g. the type of business), geographic preferences (e.g. business location ), or temporal preferences (e.g. business opening date), and others. The preference attributes required for composing a workflow are also derived from the regulatory topic hierarchy. For example, the business registration related user profile attributes include business type, location, kind and labor information. Definition 4. [User Profile] A user profile P is defined as a pair of g, P R where g is a user’s goal service g ∈ S that a user intends to initiate, and P R is a set of service preferences {p1 , p2 ..., pn }. Each pi is represented as a pair of an attribute and its value, at, v. The following is a profile for John Smith: – PJohn = open new business, { type, in corporated , name, Car Care , em kind, Autobody shop , location, 80 Mercer Street, Newark, NJ 07052 , employee number, ≥ 3 }
88
2.4
S.A. Chun, V. Atluri, and N.R. Adam
Workflow Composition Function
Definition 5. [Workflow Composition Functions] Given S, a set of services in service ontology SO, R a set of rules in rule ontology RO and P a set of user preferences, A rule selection function σ is defined as σ : (S × R × P ) → CR, where CR = {r1 , r2 , ..} ⊆ R where each ri is a pair c, a for condition c and action. Given CR from σ, workflow composition functions h and k denote task selection function and task coordination function, respectively, and are defined as follows: 1. h : CR → T , h is a function that selects a set of tasks T ∈ a(CR) where a(CR) is an action a of each rule r ∈ CR. for each t ∈ T , and P re(t) = te is precondition value is set to a trigger expression te ∈ c(CR) 2. k : CR → D, k maps all tasks T ∈ CR into dependency set D = T × T according to the coordination action a ∈ CR. Definition 6. [Workflow] A workflow W is defined as a directed graph G = (T, D) where T ⊆ S is a node set representing a set of tasks selected by composition function h and D ⊆ (T × T ) is an edge set for coordination dependencies among T , generated by composition function k. Intertask dependencies D support a variety of workflow coordination requirements. Basic types of task dependencies include control-flow dependencies which specify the flow of control based on the outcome state of a task (e.g. begin-onsuccess), value-dependencies which specify the control flow based on the output value of a task (e.g. number of employee ≥ 2), and external dependencies which specify the control flow based on certain conditions satisfied by parameters external to the workflow (e.g. task t starts at 9:00am) [2,13,3].
3
Dynamic Workflow Composition
This section presents components and an algorithm to dynamically generating a customized workflow. 3.1
Profile Gathering
User profile attributes are derived from the rule ontology and hierarchically organized. The profile attribute data required for registering a new business shown in Figure 5 consist of attributes like structure, location, name, type and employee, and so on. A leaf node represents an attribute that assumes a value. For instance, the attribute structure, can take a value from the set {incorporated, sole proprietorship, limited partnership, ...}. This hierarchical profile organization is represented as a set of profile rewrite rules (Table 1). These rules are used to gather profile information by dynamically expanding the LHS of a rule with RHS nodes and collapsing RHS nodes when all the attributes are filled with values. Figure 6 illustrates the expansion and shrinkage of the tree structure as the profile information is gathered.
Domain Knowledge-Based Automatic Workflow Generation (a)
business Registration
structure location
name
business Registration
structure location
type
(b)
employee
inc. sole ltd payroll prop. partner. manufacture service retail
89
name
type
employee
type
employee
business Registration
#of employees structure location
name
incorporated
autobody shop spray paint
restaurant
sell gas
business Registration
(c)
location
Fig. 5. Profile Attribute Hierarchy
name
type
employee
Fig. 6. Tree Structure for Profile Evaluation
Table 1. Profile Rewrite Rules P1: P2: P3: P4: P5: P6: P7: P8: P9:
business registration → structure ∧ location ∧ name ∧ type ∧ employee structure → incorporated structure → sole proprietorship location → street ∧ county ∧ state ∧ zip type → manuf acture type → service service → autobody repair autobody repair → amount(spray paint) ≥ 1/2 gallonhr ∧ sell gas employee → payroll ≥ $1000 ∧ number(employee) ≥ 1
Table 2. Service Selection and Coordination Rules R1: struct = incorporated, insert(t1) ∧ insert(t2) R2: struct = limited liabilitycompany, insert(t1) ∧ insert(t2) R3: struct = limited partnership, insert(t1) ∧ insert(t2) R5: struct = sole proprietorship, insert(t3) R7: register business, register business f or tax R8: register business, obtain f ederal employer id R9: register business, certif icate of incorporation ≺ register business f or tax R10: register business, f ein ≺ register business f or tax
3.2
Automatic Integration of Customized Workflows
Given a set of customization rules and a user profile, the customization process generates an individualized workflow, selecting only relevant component services (i.e. tasks) and obeying other constraints such as sequence ordering rules. Given the customization rule base in Tables 2, and user profile, the customization process automatically selects tasks and arranges these tasks according to the coordination rules. For instance, when a user enters incorporated as his business structure, the system finds rule R1 in Table 2, and inserts t1 and t2 into the workflow. Here, we assume t1 = register business name with the State of NJ, t2 = file original business certificate, and t3 = register business’s name with the County Clerk. If the user prefers to establish a business of sole proprietorship, then R5 fires. As a result, t3 is inserted into the workflow. As soon as a rule fires,
90
S.A. Chun, V. Atluri, and N.R. Adam
the profile evaluation tree shrinks, and the next profile attribute evaluation starts (Figure 6). This process repeats until all the attributes of the profile tree acquire values and are used to fire appropriate rules. The end result of this process is a workflow. In addition to the tasks selected based on preferences, there exist obligatory tasks required for all users. For instance, register-business-for-tax is required for all applicants. These obligatory tasks are inserted by rules such as R7 and R8 in Table 2. All the tasks in the workflow are then subjected to the inter-task sequencing rules to ensure the coordination requirements, such as R9 and R10 in Table 2. For instance, rule R9 states that if the process is register business, then the task obtain a certificate of incorporation should be finished before register business for tax. The following algorithm generates an inter-agency workflow given a user’s request for a service defined in 2.1 and 2.2. q g g. It uses the ontologies g Algorithm 1.[Generate Customized workflow] Input: g: user’s desired service Output: W=(T, D) W=workflow with set of tasks T and dependency D BEGIN T = ∅, D = ∅ S =has-component(g) /* identify all component services for G */ R = ∅ /* set of rules applicable to services in S */ For each si ∈ S { /* identify regulations and rules related to S */ reg = implement(si ) r = related-to(reg) R=R∪r } For each r = (c, a) ∈ S { /* add obligatory service in workflow tasks T */ if (c(r) = ∅ { T = T ∪ t ∈ a(r) } For each ti ∈ T and r = (c, a) ∈ R { /* add dependencies among tasks */ if a(r) = order(ti , tj ) { D = D ∪ (ti , tj ) P re(tj ) = bs R=R−r } S = S -T PR = gather-profile∗ (g) /* gather relevant user profile */ While(PR = empty ) Do { /* while preference list is not empty */ Given p ∈ P R For each r = (c, a) ∈ R { /* evaluate preference rules */ if (eval(c, p) = true) { if a = insert(ti ) { /* add a task into workflow */ T = T ∪ ti P re(ti ) = te ∈ c } /* add trigger expression as precondition for ti */ For each ti ∈ T { /* add dependency */ if a = order(ti , tj ) { D = D ∪ (ti , tj ) P re(tj ) = P re(tj ) ∧ bs R = R − r }}}} PR = PR − p S = S − ti } W = (T, D) } END ∗ We assume the function gather-profile returns a set of attribute and value sets required for a service g.
The algorithm first identifies all the candidate component tasks based on the user requested service, g, as specified in the component service ontology. Second, it identifies applicable regulatory rules that are used to select obligatory and other tasks and determine dependencies among tasks. Selected tasks are composed according to the order related rules. The output of this algorithm is the customized composite workflow W = (T, D) for the requested service g.
Domain Knowledge-Based Automatic Workflow Generation
3.3
91
Prototype System
We have developed a prototype system1 of generating customized workflows for the NJ State government [1]. Specifically, we have built an inter-agency service, called “a new business registration” process that requires information and coordination of various State agencies including Department of Revenue, Division of Commercial Recording, Division of Community Affairs, Department of Environmental Protection, Department of Public Health and Safety and others. The prototype system gathers the user profile information through an interview session, which is guided by interview rules which are based on case analysts’ expert knowledge, i.e. ontology, such that only questions relevant to a specific business of the user are asked. The user profile gathered at the interview session and the business related regulations database are matched to generate a customized workflow. Web-based client applications for entrepreneurs include (1) a GIS-based business location service; (2) an interview interface; and (3) a workflow interface.
4
Related Work
Industry standards and research efforts such as WSDL, UDDI, ebXML, DAMLS, describing the semantics and metadata of web services, facilitate discovery and access of services, but they go only half way toward automating integration of services. The automation of service integration requires the selection and control knowledge that specifies what services are needed in certain circumstances. This knowledge of service composition and selection is left to domain experts to define manually, or for program developers to combine with the application code. Our approach of utilizing a domain model provides the capability to automate the integration and customization of workflow design. Platforms for developing composite e-services proposed in eFlow [6], E-speak [11], and CMI (Collaboration Management Infrastructure) [15] are compositional models, similar to our approach. However, their design of e-services (compositional process) is still manual, while our approach uses a workflow definition model where the workflow task selection and ordering knowledge is explicitly modeled for automating the generation of the workflow definition.
5
Conclusions and Future Work
We have presented a formal model for automatic workflow generation that uses domain knowledge represented as service ontology, regulatory ontology and a user profile. We have presented the composition algorithm and data structures used for automatic generation of the workflow. This framework can be easily adopted to process definitions in other application domains where process design needs to obey company policies and other constraints. Our future work includes semiautomatically constructing the ontology of regulations and extracting business rules from documents. We also plan to address the workflow analysis and verification which provide validation of the automatically generated workflow. The 1
http://cimic.rutgers.edu/dgov/demos1.html
92
S.A. Chun, V. Atluri, and N.R. Adam
dynamic modifications of workflow definitions to run-time changes in regulations, user profiles, or services are also under investigation.
References 1. N. Adam, F. Artigas, V. Atluri, S. Chun, S. Colbert, M. Degeratu, A. Ebeid, V. Hatzivassiloglou, R. Holowczak, O. Marcopolus, P. Mazzoleni, and W. Rayner. E-government: Human centered systems for business services. In The First National Conference on Digital Government, pages 48–55, May 2001. 2. Nabil R. Adam, Vijayalakshmi Atluri, and Wei-Kuang Huang. Modeling and Analysis of Workflows using Petri nets. Journal of Intelligent Information Systems, Special Issue on Workflow and Process Management,, 10(2), March 1998. 3. V. Alturi, S. Chun, and Pietro Mazzoleni. A Chinese Wall Secuity Model for Decentralized Workflow Systems. In Eighth ACM Conference on Computer and Communications Security (CCS-8), Philadelphia, Pennsylvania, USA, November 2001. 4. A. Bouguettaya, A. Elmagarmid, B. Madjahed, and M. Quzzani. A web-based architecture for government databases and services. In The First National Conference on Digital Government, pages 56–59, May 2001. 5. F. Casati, S. Ceri, B. Pernici, and G. Pozzi. Conceptual Modeling of Workflows. In Proceedings of the 1995 OOER Conference, December 1995. 6. Fabio Casati, Ski Ilnicki, and Li-Jie Jin. eflow: a platform for developing and managing composite eservices. Technical report, HP Laboratories Palo Alto, March 2000. 7. Dimitrios Georgakopoulos, Mark Hornick, and Amit Sheth. An Overview of Workflow Management: From Process Modeling to Workflow Automation Infrastructure. Distributed and Parallel Databases, pages 119–153, 1995. 8. T.R. Gruber. Toward principles for the design of ontologies used for knowledge sharing. In N. Guarino and R. Poli, editors, Proceedings of International Workshop on Formal Ontology, Padova, Italy, 1993. 9. C. S. Han. Computer Models and Methods for a Disabled Access Analysis Design Enviroment. PhD thesis, Department of Civil and Environmental Engineering, Stanford University, June 2000. 10. R. Holowczak, S. Chun, F. Artigas, and V. Atluri. Customized geospatial workflows for e-government services. In ACM-GIS 2001, The Ninth ACM International Symposium on Advances in Geographic Information Systems, Atlanta, GA, USA, November 2001. 11. Alan Karp. E-speak e-xplained. Technical report, HP Laboratories Palo Alto, July 2000. 12. Shawn Kerrigan, Gloria Lau, Liang Zhou, Gio Wiederhold, and Kincho H. Law. Information infrastructure for regulation management and compliance checking. In The First National Conference on Digital Government, pages 167–170, May 2001. 13. Marek Rusinkiewicz and Amit Sheth. Specification and Execution of Transactional Workflows. In W. Kim, editor, Modern Database Systems: The Object Model, Interoperability, and Beyond. Addison-Wesley, 1994. 14. Asim Sadiq and Mria Orlowska. Applying a generic conceptual workflow modeling technique to document workflows. In The second Australian Document Computing Symposium, Melbourne, Australia, April 1997.
Domain Knowledge-Based Automatic Workflow Generation
93
15. H. Schuster, D. Georgakopoulos, A. Cichocki, and D. Baker. Modeling and composing service-based and reference process-based multi-enterprise processes. In Proceedings of the Inernational Conference on Advanced Information Systems Engineering (CAiSE), Stockholem, Sweden, June 2000.
93
R. Cicchetti et al. (Eds.): DEXA 2002, LNCS 2453, pp. 93–102, 2002. c Springer-Verlag Berlin Heidelberg 2002
94
M. Cai et al.
A Comparison of Alternative Encoding Mechanisms for Web Services
95
96
M. Cai et al.
A Comparison of Alternative Encoding Mechanisms for Web Services
97
98
M. Cai et al.
A Comparison of Alternative Encoding Mechanisms for Web Services
99
100
M. Cai et al.
A Comparison of Alternative Encoding Mechanisms for Web Services
101
102
M. Cai et al.
Capturing Semantics in HTML Documents Mengchi Liu School of Computer Science, Carleton University Ottawa, Ontario K1S 5B6, Canada [email protected]
Abstract. Most documents available over the web confirm to the HTML specification. They are intended to be human readable through a web browser and thus are constructed following some common conventions. Based on such common conventions, the Conceptual Model for HTML was proposed recently to automatically capture the hierarchical structure within web documents. However, certain key semantic information about the contents in the documents, which are obvious to human, are often omitted. As a result, web data processing, manipulation and integration are still quite difficult. In this paper, we discuss how to extend the Conceptual Model for HTML to capture the intended semantics of the HTML documents. We show that with the new constructs introduced, using an Intelligent Wrapper, and limited human interaction, semantics can be transferred from human into the Extended Conceptual Model so that further meaningful processing, manipulation and integration of web documents become possible.
1
Introduction
With the ever-growing popularity of the World Wide Web, the Internet has become a major source for various kinds of information. How to make meaningful processing, manipulation and integration and provide semantically correct answers has become a major research issue. Several different approaches have been exploited to retrieve information from the web: using information retrieval techniques to identify predefined concepts, extracting structure in documents based on formatting information [1,9,11,12,16], or focusing on reconstructing the web documents using wrappers, and developing powerful query languages [1,2,3,6,13]. Other proposals use database techniques and tools to manage web data, as found in [2,7,10], which discuss approaches based on ER models, SQL, relational algebra and other database techniques. For a survey see [8]. These various approaches reveal the same fact: the need for capturing the structure and semantics about the information over the web. Most web documents are human-oriented. They don’t support explicit structure and semantics within the documents. However, they normally follow some common conventions and often exhibit some hierarchical structure. Based on such common conventions, the Conceptual Model for HTML was proposed to automatically capture R. Cicchetti et al. (Eds.): DEXA 2002, LNCS 2453, pp. 103–112, 2002. c Springer-Verlag Berlin Heidelberg 2002
104
M. Liu
the hierarchical structure within the web document with a few simple constructs in [13]. It provides a simple way to describe web documents at a high level close to human visualization. However it is still not sufficient as a basis for further semantics-based processing as certain key semantic information about the contents, which are obvious to human, are often omitted in the documents. The constructs provided in the Conceptual Model are too simple and the semantic information if any cannot be expressed naturally. In particular, tuple-like information and lists are all represented as lists, which causes some ambiguity in both object notion and relationships between objects. Furthermore, table is the most used construct in HTML documents for presenting complicated information in a simply way. However, in the Conceptual Model, tables are converted into nested lists, which can be quite complicated, but neither natural nor informative. In this paper, we discuss how to extend the Conceptual Model for HTML to capture both structural and semantic information in HTML documents. We show that with the new constructs introduced, using an Intelligent Wrapper, and limited human interaction, semantics can be transferred from human into the Extended Conceptual Model so that further meaningful processing, manipulation and integration of web documents become possible. The rest of the paper is organized as follows. In section 2, we discuss how to extend the Conceptual Model. In section 3, we discuss how the Extended Conceptual Model is used in our Web Query and Inference System. Finally, we summarize and point out further research issues.
2
Extending the Conceptual Model for HTML
The purpose of models is not to fit the data but to sharpen the questions [15]. In databases we distinguish attributes from values and we simply use attributes to query the database even though they are deeply nested. For web documents, if we can identify attributes, then we can query web documents in the same way. This is the idea behind the Conceptual Model for HTML in [13], which provides a simple way to represent the complex hierarchical structure within web documents using attribute and value pairs, lists, etc. However, the objects in the Conceptual Model may still not be meaningful. Consider a simplified web document shown in Figure 1. It is about the people in the Department of Computer Science and is quite meaningful to human. It is represented in the Conceptual Model as follows: People ⇒{{Anne, Lecturer, B200, {Gary,Professor,B210, {John, Professor,B190, {Mark, Student, B170, {Mary, Staff, B130, {Tony,Professor,B150,
520-2500, 520-2600, 520-2400, 520-2300, 520-2100, 520-2200,
[email protected]} [email protected] }, [email protected] }, [email protected]}, [email protected]}, [email protected] }}
where People is an attribute and the list after ⇒ is its value. The list consists of four nested lists as well.
Capturing Semantics in HTML Documents
105
Fig. 1. Sample Document
In databases, lists are normally used to represent sequences of similar objects, while tuples are normally used to represent different aspects of similar objects. Using lists for tuples loses some semantic information and is an obvious deficiency of the model. The above object is better represented as follows: People ⇒{[Anne, Lecturer, B200, 520-2500, [email protected]], [Gary, Professor, B210, 520-2600, [email protected] ], [John, Professor, B190, 520-2400, [email protected] ], [Mark, Student, B170, 520-2300, [email protected]], [Mary, Staff, B130, 520-2100, [email protected]], [Tony, Professor, B150, 520-2200, [email protected] ]} In this way, we know at least that the data items in a tuple represent some aspect of a person and there are six people altogether and this view is close to what we see in Figure 1. Thus, our first extension is to add tuple construct to the Conceptual Model. For another example, consider part of the latest DBLP bibliography server of Michael Ley at www.informatik.uni-trier.de/∼ ley/db shown in Figure 2. Its representation with the new tuple construct is shown in Figure 3 where a1 , b1 , etc. are simplified URLs to fit in the paper. Consider again the sample document in Figure 1. If we want to find all people using a query, then we can use the attribute People to get what we want. In other words, People is the semantic information that describes the contents in the document and can be used to interpret or query the document. However, if we want information about individual person such as office, phone number, or email address, we do not have any semantic information to use in our queries.
106
M. Liu
Fig. 2. DBLP Bibliography Web Page
What we can do is to transfer human knowledge to the Conceptual Model. However, the Conceptual Model does not have proper constructs that can be used to represent such semantic information. In databases, a list of tuples is a table and the schema of the table is the most important part for table queries and is normally separated from the table. What we are lacking in the above example is just the schema. Unlike in databases, if we want to add schema for a table, the schema should not be separated from the table. Instead, the schema should be kept together with the table so that the system can use it to interpret data, just like the element tags and attribute names in XML [4]. Thus, our second extension to the Conceptual Model is the notion of table. We use {...}T to represent a table of tuples and the first tuple is the schema of the table and we use [...]S to represent this tuple so that it can be distinguished from ordinary tuples. With these new constructs and knowledge transfer from human, the above object can be represented meaningfully as follows:
Capturing Semantics in HTML Documents
107
[
... Title ⇒DBLP Bibliography, Search ⇒ {Authora1 , Titlea2 , Advanceda3 , Home Page Searcha4 }, Bibliographies ⇒ [ Conferencesb1 ⇒{SIGMODb11 , VLDBb12 , PODSb13 , ...}, Journalsb2 ⇒{CACMb21 , TODSb22 , TOISb23 , ... }, Seriesb3 ⇒{LNCS/LNAIb41 , DISDBISb42 }, Books ⇒{Collectionsb51 , DB Textbookb52 }, By Subjectsb4 ⇒{Database Systemsb61 , Logic Progb62 , IRb63 }, ] Full Text ⇒ACM SIGMOD Anthologyc1 , Reviews ⇒ACM SIGMOD Digital Reviewc2 , Links ⇒[ Research Groups ⇒{Database Systemsd1, Logic Programmingd2}, Computer Science Organizatione1 ⇒{ ACMe11 (DLe12 , SIGMODe13 , SIGIR e14 ), IEEE Computer Societye15 (DLe16 )} Related Services f1 ⇒{ CoRRf11 , Research Indexf12 , NZ-DLf13 , CS BibTexf14 , HBPf15 , Virtual Library f16 } ] ... ] Fig. 3. DBLP Web Object with Simplified URLs
People ⇒{[Name, Position, [Anne, Lecturer, [John, Professor, [Gary, Professor, [Mark, Student, [Mary, Staff, [Tony, Professor,
Office B200, B190, B210, B170, B130, B150,
, Phone, 520-2500, 520-2400, 520-2600, 520-2300, 520-2100, 520-2200,
Email ]S [email protected]], [email protected] ], [email protected] ], [email protected]], [email protected]],