WAIM 2007 International Workshops: DBMAN 2007, WebETrends 2007, PAIS 2007 and

Lecture Notes in Computer Science Commenced Publication in 1973 Founding and Former Series Editors: Gerhard Goos, Juris Hartmanis, and Jan van Leeuwen

Editorial Board David Hutchison Lancaster University, UK Takeo Kanade Carnegie Mellon University, Pittsburgh, PA, USA Josef Kittler University of Surrey, Guildford, UK Jon M. Kleinberg Cornell University, Ithaca, NY, USA Friedemann Mattern ETH Zurich, Switzerland John C. Mitchell Stanford University, CA, USA Moni Naor Weizmann Institute of Science, Rehovot, Israel Oscar Nierstrasz University of Bern, Switzerland C. Pandu Rangan Indian Institute of Technology, Madras, India Bernhard Steffen University of Dortmund, Germany Madhu Sudan Massachusetts Institute of Technology, MA, USA Demetri Terzopoulos University of California, Los Angeles, CA, USA Doug Tygar University of California, Berkeley, CA, USA Moshe Y. Vardi Rice University, Houston, TX, USA Gerhard Weikum Max-Planck Institute of Computer Science, Saarbruecken, Germany

4537

Kevin Chen-Chuan Chang Wei Wang Lei Chen Clarence A. Ellis Ching-Hsien Hsu Ah Chung Tsoi Haixun Wang (Eds.)

Advances in Web and Network Technologies, and Information Management APWeb/WAIM 2007 International Workshops: DBMAN 2007, WebETrends 2007, PAIS 2007 and ASWAN 2007 Huang Shan, China, June 16-18, 2007 Proceedings

13

Volume Editors Kevin Chen-Chuan Chang University of Illinois at Urbana-Champaign, USA E-mail: [email protected] Wei Wang, University of New South Wales, Australia E-mail: [email protected] Lei Chen, Hong Kong University, Hong Kong E-mail: [email protected] Clarence A. Ellis, University of Colorado at Boulder, USA E-mail: [email protected] Ching-Hsien Hsu, Chung Hua University, Taiwan E-mail: [email protected] Ah Chung Tsoi, Monash University, Australia E-mail: [email protected] Haixun Wang, T. J. Watson Research Center, USA E-mail: [email protected]

Library of Congress Control Number: 2007928328 CR Subject Classification (1998): H.2-5, C.2, I.2, K.4, J.1 LNCS Sublibrary: SL 3 – Information Systems and Application, incl. Internet/Web and HCI ISSN ISBN-10 ISBN-13

0302-9743 3-540-72908-9 Springer Berlin Heidelberg New York 978-3-540-72908-2 Springer Berlin Heidelberg New York

This work is subject to copyright. All rights are reserved, whether the whole or part of the material is concerned, specifically the rights of translation, reprinting, re-use of illustrations, recitation, broadcasting, reproduction on microfilms or in any other way, and storage in data banks. Duplication of this publication or parts thereof is permitted only under the provisions of the German Copyright Law of September 9, 1965, in its current version, and permission for use must always be obtained from Springer. Violations are liable to prosecution under the German Copyright Law. Springer is a part of Springer Science+Business Media springer.com © Springer-Verlag Berlin Heidelberg 2007 Printed in Germany Typesetting: Camera-ready by author, data conversion by Scientific Publishing Services, Chennai, India Printed on acid-free paper SPIN: 12073054 06/3180 543210

APWeb/WAIM 2007 Workshop Chair’s Message

As an important part of the joint APWeb/WAIM 2007 conference, continuing both conferences’ traditional excellence on theoretical and practical aspects of Web-based information access and management, we were pleased to have four workshops included in this joint conference. This proceedings volume compiles, the technical papers selected for presentation at the following workshops, held at Huang Shan (Yellow Mountains), China, June 16 – 18, 2007. 1. International Workshop on DataBase Management and Application over Networks (DBMAN 2007) 2. International Workshop on Emerging Trends of Web Technologies and Applications (WebETrends 2007) 3. International Workshop on Process Aware Information Systems (PAIS 2007) 4. International Workshop on Application and Security service in Web and pervAsive eNvirionments (ASWAN 2007) These four workshops were selected from a public call-for-proposals process. The workshop organizers put a tremendous amount of effort into soliciting and selecting research papers with a balance of quality, novelty, and application relevance. We asked all workshops to follow a rigid paper selection process, including the procedure to ensure that any Program Committee members (including workshop Program Committee Chairs) were excluded from the paper review process of any papers they were involved in. A requirement about the overall paper acceptance ratio was also imposed on all the workshops. I am very grateful to Jeffrey Xu Yu, Xuemin Lin, Zheng Liu, Wei Wang and many other people for their great effort in supporting the conference organization. I would like to take this opportunity to thank all workshop organizers and Program Committee members for their great effort in putting together the workshop program. I hope you enjoy these proceedings in. June 2007

Kevin C. Chang

International Workshop on DataBase Management and Application over Networks (DBMAN 2007) Chairs’ Message

With the increasing ubiquity of personal computing devices, such as mobile phones and PDAs, and the increasing deployment of sensor networks, new distributed applications are developed over networked databases posing interesting challenges. The 1st International Workshop on Database Management and Applications over Networks (DBMAN 2007) was held in Huanshan, China on June 18, 2007, in conjunction with the WAIM/APWeb 2007 conference. It aimed to bring together researchers in different fields related to database management and application over networks and to provide a forum where researchers and practitioners could share and exchange their knowledge and experience. In response to the call for papers, DBMAN attracted 105 submissions. The submissions are highly diversified, coming from Australia, China, France, Japan, Korea, New Zealand, Norway, Spain, Thailand, Taiwan and the USA, resulting in an international final program. All submissions were peer reviewed by at least two Program Committee members. The Program Committee selected 14 full papers and 14 short papers for inclusion in the proceedings. The competition was keen, with an overall acceptance rate of around 27%. The accepted papers covered a wide range of research topics and novel applications on database management over networks. The workshops would not have been successful without the help of many organizations and individuals. First, we would like to thank the Program Committee members and external reviewers for evaluating the assigned papers in a timely and professional manner. Next, the tremendous efforts put forward by members in the Organization Committee of WAIM/APWeb in accommodating and supporting the workshops are appreciated. Of course, without support from the authors with their submissions, the workshop would not have been possible. June 2007

Lei Chen ¨ M. Tamer Ozsu

International Workshop on Emerging Trends of Web Technologies and Applications (WebETrends 2007) Chairs’ Message

The 1st Workshop on Emerging Trends of Web Technologies and Applications was held in Huangshan, China, on June 16, 2007, in conjunction with APWeb/WAIM 2007: the Ninth Asia-Pacific Web Conference (APWeb) and the Eighth International Conference on Web-Age Information Management (WAIM). The explosive growth of the World Wide Web is revolutionizing how individuals, groups, and communities interact and influence each other. It is well recognized that the Web has transformed everyday life; however, its deepest impact in science, business, and society is still in the making. As technologies mature and communities grow, a new world that is fundamentally revolutionary is just beginning. The goal of this workshop is to bring together researchers to share visions for the future World Wide Web. The workshop solicted papers in a wide range of fields with a focus on emerging Web technologies and applications. We welcome innovative ideas from traditional topics such as Web wrapping, Internet crawling, search engine optimization, current hot areas such as Wiki, blogosphere, social network, as well as promising future directions such as Web services and the Semantic Web. Our aim is to promote the awareness of the potential impact of the Web, discuss research directions and agendas, share experience and insights, and build a joint community across disciplines for creating enabling technologies for the future Web. The workshop received 40 submissions from many countries and regions. All submissions were peer reviewed by at least two Program Committee members. The Program Committee selected 12 full papers for inclusion in the proceedings. We are grateful to members of the Program Committee who contributed their expertise and ensured the high quality of the reviewing process. We are thankful to the Workshop Chair Kevin Chang and the PC Chair Jeffrey Xu Yu for their support.

June 2007

Haixun Wang Chang-shing Perng

International Workshop on Process Aware Information Systems (PAIS 2007) Chairs’ Message

A process-aware information system (PAIS) is a software system that manages and executes operational processes involving people, applications, and/or information sources on the basis of an explicit imbedded process model. The model is typically instantiated many times, and every instance is typically handled in a predefined way. Thus this definition shows that a typical text editor is not process aware, and likewise a typical e-mail client is not process aware. In both of these examples, the software is performing a task, but is not aware that the task is part of a process. Note that the explicit representation of process allows automated enactment, automated verification, and automated redesign, all of which can lead to increased organizational efficiency. The strength and the challenge of PAISs are within the CSCW domain where PAISs are used by groups of people to support communication, coordination, and collaboration. Even today, after years of research and development, PAISs are plagued with problems and pitfalls intertwined with their benefits. These problems are frequently elusive and complex due to the fact that “PAISs are first and foremost people systems.” Conclusively, we were able to address some of the people issues and frameworks for research in this domain through the workshop, as we expected. Although the workshop was the first in the series, we received a reasonable number of submissions, 60 papers from about 20 regions and countries. All submissions were peer reviewed by at least two or three Program Committee members and five external reviewers. After a rigorous review process, the Program Committee selected 8 full papers and 11 short papers among the 60 submissions. The acceptance rates of full paper and short paper submissions are 13.3% and 18.3%, respectively. Also, the accepted papers covered a wide range of research topics and novel applications related to PAISs. We are grateful to the Kyonggi University and the University of Colorado at Boulder for sponsoring the workshop and would like to express thanks to the research members of the CTRL/CTRG research groups for developing the workshop’s Web site and review-processing system. We would also like to thank all authors who submitted papers and all the participants in the workshop program. We are especially grateful to members of the Program Committee who contributed their expertise and ensured the high quality of the reviewing process. We are thankful to the APWeb/WAIM organizers for their support and local arrangement. June 2007

Clarence A. Ellis Kwanghoon Kim

International Workshop on Application and Security Service in Web and Pervasive Environments (ASWAN 2007) Program Chairs’ Message

We are proud to present the proceedings of the 2007 International Workshop on Application and Security Service in Web and Pervasive Environments, held at HuangShan, China during June 16–18, 2007. Web and pervasive environments (WPE) are emerging rapidly as an exciting new paradigm including ubiquitous, Web, grid, and peer-to-peer computing to provide computing and communication services any time and anywhere. In order to realize their advantages, security services and applications need to be suitable for WPE. ASWAN 2007 was intended to foster the dissemination of state-ofthe-art research in the area of secure WPE including security models, security systems and application services, and novel security applications associated with its utilization. The aim of ASWAN 2007 was to be the premier event on security theories and practical applications, focusing on all aspects of Web and pervasive environments and providing a high-profile, leading-edge forum for researchers and engineers alike to present their latest research. In order to guarantee high-quality proceedings, we put extensive effort into reviewing the scientific papers. We received 61 papers from Korea, China, Hong Kong, Taiwan and the USA, representing more than 50 universities or institutions. All submissions were peer reviewed by two to three Program or Technical Committee members or external reviewers. It was extremely difficult to select the presentations for the workshop because there were so many excellent and interesting ones. In order to allocate as many papers as possible and keep the high quality of the workshop, we decided to accept 16 papers for oral presentations. We believe all of these papers and topics will not only provide novel ideas, new results, work in progress and state-of-the-art techniques in this field, but will also stimulate future research activities. This workshop would not have been possible without the support of many people to make it a success. First of all, we would like to thank the Steering Committee Chair, Laurence T. Yang, and Jong Hyuk Park for nourishing the workshop and guiding its course. We thank the Program Committee members for their excellent job in reviewing the submissions and thus guaranteeing the quality of the workshop under a very tight schedule. We are also indebted to the members of the Organizing Committee. Particularly, we thank Byoung-Soo Koh, Jian Yang, Xiaofeng Meng, Yang Xiao and Yutaka Kidawara for their devotions

X

Preface

and efforts to make this workshop a real success. Finally, we would like to take this opportunity to thank all the authors and participants for their contributions, which made ASWAN 2007 a grand success. June 2007

Laurence T. Yang Sajal K. Das Eung Nam Ko Ching-Hsien Hsu Djamal Benslimane Young Yong Kim

Organization

International Workshop on DataBase Management and Application over Networks (DBMAN 2007) Program Co-chairs Lei Chen, Hong Kong University of Science and Technology, Hong Kong, China ¨ M. Tamer Ozsu, University of Waterloo, Waterloo, Canada

Program Committee Gustavo Alonso, ETH, Switzerland Luc Bouganim, INRIA, France Klemens B¨ohm, Universit¨ at Karlsruhe (TH), Germany Ilaria Bartolini, University of Bologna, Italy Ahmet Bulut, Citrix Systems, USA Selcuk Candan, Arizona State University, USA Yi Chen, Arizona State University, USA Reynold Cheng, Hong Kong Polytechnic University, China Mitch Cherniack, Brandeis University, USA Khuzaima Daudjee, University of Waterloo, Canada Ada Waichee Fu, Chinese University of Hong Kong, China Bj¨orn TH´ or J´ onsson, Reykjavik University, Iceland Wang-Chien Lee, Penn State University, USA Chen Li, University of California, Irvine, USA Mingjin Li, Microsoft Research Asia, China Alexander Markowetz, Hong Kong University of Science and Technology, China Vincent Oria, New Jersey Institute of Technology, USA Sule Gunduz Oguducu, Istanbul Technical University, Turkey Jian Pei, Simon Fraser University, Canada Peter Triantafillou, University of Patras, Greece Anthony Tung, National University of Singapore, Singapore ¨ ur Ulusoy, Bilkent University, Turkey Ozg¨ Patrick Valduriez, INRIA, France Jari Veijaleinen, University of Jyvaskyla, Finland JianLiang Xu, Hong Kong Baptist University, China

XII

Organization

External Reviewers Yu Li, Jinchuan Chen, Xiang Lian, Yingyi Bu, Qiang Wang, Yingying Tao, Huaxin Zhang, Oghuzan Ozmen, Lukasz Golab, Weixiong Rao, Shaoxu Song, Rui Li, Lei Zou, Yongzhen Zhuang, Xiaochun Yang, Bin Wang.

International Workshop on Emerging Trends of Web Technologies and Applications (WebETrends 2007) Program Co-chairs Haixun Wang, IBM T. J. Watson Research Center, USA Chang-shing Perng, IBM T. J. Watson Research Center, USA

Program Committee Jeff Wei-shinn Ku, University of Southern California, USA Yijian Bai, University of California, Los Angeles, USA Jian Yin, IBM T. J. Watson Research Center, USA Tao Li, Florida International University, USA Hui Xiong, Rutgers University, USA Charles Perng, IBM T. J. Watson Research Center, USA Haixun Wang, IBM T. J. Watson Research Center, USA

International Workshop on Process Aware Information Systems (PAIS 2007) Program Co-chairs Clarence A. Ellis, University of Colorado at Boulder, USA Kwang-Hoon Kim, Kyonggi University, Korea

Program Committee Hayami Haruo, Kanagawa Institute of Technology, Japan Jintae Lee, University of Colorado at Boulder, USA George Wyner, Boston University, USA Jorge Cardoso, University of Madeira, Portugal Yang Chi-Tsai, Flowring Technology, Inc., Taiwan Michael zur Muehlen, Stevenson Institute of Technology, USA Dongsoo Han, Information and Commnications University, Korea Ilkyeun Ra, University of Colorado at Denver, USA Taekyou Park, Hanseo University, Korea Joonsoo Bae, Chonbuk National University, Korea

Organization

XIII

Yoshihisa Sadakane, NEC Soft, Japan Junchul Chun, Kyonggi University, Korea Luis Joyanes Aguilar, Universidad Pontificia de Salamanca, Spain Tobias Rieke, University of Muenster, Germany Modrak Vladimir, Technical University of Kosice, Slovakia Haksung Kim, Dongnam Health College, Korea Yongjoon Lee, Electronics and Telecommunications Research Institute, Korea Taekyu Kang, Electronics and Telecommunications Research Institute, Korea Jinjun Chen, Swinburne University of Technology, Australia Zongwei Luo, The University of Hong Kong, China Peter sakal, Technical University Bratislava, Slovakia Sarka Stojarova, Univeristy of Brno, Czech Republic Boo-Hyun Lee, KongJu National University, Korea Jeong-Hyun Park, Electronics and Telecommunications Research Institute, Korea Yanbo Han, Chinese Academy of Sciences, China Jacques Wainer, State University of Campinas, Brazil Aubrey J. Rembert, University of Colorado at Boulder, USA

External Reviewers Aubrey Rembert, Hyongjin Ahn, Minjae Park, Jaekang Won, Hyunah Kim

Workshop Liaison Taewook Kim

International Workshop on Application and Security Service in Web and Pervasive Environments (ASWAN 2007) Steering Chair Laurence T. Yang, St. Francis Xavier University, Canada

General Co-chairs Sajal K. Das, University Texas at Arlington, USA Eung Nam Ko, Baekseok University, Korea

Program Co-chairs Ching-Hsien Hsu, Chung Hua University, Taiwan Djamal Benslimane, University Claude Bernard, France Young Yong Kim, Yonsei University, Korea

XIV

Organization

Publicity Co-chairs Jian Yang, Macquarie University, Australia Xiaofeng Meng, Renmin University of China, China Yang Xiao, University of Alabama, USA Yutaka Kidawara, NICT, Japan

Web Management Chair Byoung-Soo Koh, DigiCAPS Co., Ltd, Korea

Program Committee Andrew Kusiak, The University of Iowa, USA Anna Cinzia Squicciarini, Purdue University, USA Antonio Coronato, SASIT-CNR, Italy Apostolos N. Papadopoulos, Aristotle University, Greece Aris M. Ouksel, The University of Illinois at Chicago, USA Barbara Catania, Universit` a di Genova, Italy Cho-Li Wang, The University of Hong Kong, China Christoph Bussler, Cisco Systems, USA Christophides Vassilis, Foundation for Research and Technology-Hellas, Greece Claudio Sartori, Universit` a di Bologna, Italy Deok Gyu Lee, ETRI, Korea Dimitris Papadias, Hong Kong University of Science and Technology, China Do van Thanh, NTNU, Norway Dongseop Kwon, Myungji University, Korea Emmanuelle Anceaume, IRISA, France Evi Syukur, Monash University, Australia Fangguo Zhang, Sun Yat-sen University, China Gianluca Moro, University of Bologna, Italy Huafei Zhu, Institute for Infocomm Research, Singapore Ilsun You, Korean Bible University, Korea Javier Garcia Villalba, Complutense University of Madrid, Spain Jong Hyuk Park, Hanwha S&C Co., Ltd., Korea Jean-Henry Morin, Korea University, Korea Karl M. Goeschka, Vienna University of Technology, Austria Katsaros Dimitrios, Aristotle University, Greece Marco Aiello, University of Trento, Italy Massimo Esposito, ICAR-CNR, Italy Massimo Poncino, Politecnico di Torino, Italy Mirko Loghi, Politecnico di Torino, Italy Naixue Xiong, JAIST, Japan Nicolas Sklavos, Technological Educational Institute of Messolonghi, Greece Ning Zhang, University of Manchester, UK

Organization

Paolo Bellavista, University of Bologna, Italy Paris Kitsos, Hellenic Open University, Greece Qi Shi, Liverpool John Moores University, UK Rodrigo Fernandes de Mello, University of Sao Paulo, Brazil Sheng-De Wang, National Taiwan University, Taiwan Stavros Christodoulakis, MUSIC/TUC, Crete, Greece Tetsu Iwata, Nagoya University, Japan Tom Gross, University of Weimar, Germany Tore Jonvik, Oslo Unversity College, Norway Traore Jacques RD-MAPS-CAE, France Telecom R&D, France Vesna Hassler, European Patent Office, Austria Weisong Shi, Wayne State University, USA Wenfei Fan, University of Edinburgh, UK Yeong Deok Kim, Woosong University, Korea Klaus R. Dittrich, Universit¨ at Zurich, Switzerland Trevor Jim, AT&T Labs Research, USA Wen Ouyang, Chung Hua University, Taiwan

XV

Table of Contents

International Workshop on DataBase Management and Application over Networks (DBMAN 2007) Information Access and Dissemination I Cost Framework for a Heterogeneous Distributed Semi-structured Environment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Tianxiao Liu, Tuyˆet Trˆ am Dang Ngoc, and Dominique Laurent

1

Cost-Based Vertical Fragmentation for XML . . . . . . . . . . . . . . . . . . . . . . . . Sven Hartmann, Hui Ma, and Klaus-Dieter Schewe

12

Efficiently Crawling Strategy for Focused Searching Engine . . . . . . . . . . . . Liu Huilin, Kou Chunhua, and Wang Guangxing

25

QoS-Guaranteed Ring Replication Management with Strong Consistency . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Wei Fu, Nong Xiao, and Xicheng Lu

37

A Mobile Database Sharing Protocol to Increase Data Availability in Mobile Environments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Hien Nam Le and Mads Nyg˚ ard

50

Data Mining Mining Recent Frequent Itemsets over Data Streams with a Time-Sensitive Sliding Window . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Long Jin, Duck Jin Chai, Jun Wook Lee, and Keun Ho Ryu

62

Mining Web Transaction Patterns in an Electronic Commerce Environment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Yue-Shi Lee and Show-Jane Yen

74

Mining Purpose-Based Authorization Strategies in Database System . . . . Jie Song, Daling Wang, Yubin Bao, Ge Yu, and Wen Qi

86

RSP-DS: Real Time Sequential Pattern Analysis over Data Streams . . . . Ho-Seok Kim, Jae-Jyn Shin, Yong-Il Jang, Gyoung-Bae Kim, and Hae-Young Bae

99

XVIII

Table of Contents

Sensor, P2P, and Grid Networks I Investigative Queries in Sensor Networks . . . . . . . . . . . . . . . . . . . . . . . . . . . . Madhan K. Vairamuthu, Sudarsanan Nesamony, Maria E. Orlowska, and Shazia W. Sadiq A Taxonomy-Based Approach for Constructing Semantics-Based Super-Peer Networks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Baiyou Qiao, Guoren Wang, and Kexin Xie A Comparative Study of Replica Placement Strategies in Data Grids . . . Qaisar Rasool, Jianzhong Li, George S. Oreku, Ehsan Ullah Munir, and Donghua Yang A Workload Balancing Based Approach to Discourage Free Riding in Peer-to-Peer Network . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Yijiao Yu and Hai Jin QoS-Based Services Selecting and Optimizing Algorithms on Grid . . . . . . Qing Zhu, Shan Wang, Guorong Li, Guangqiang Liu, and Xiaoyong Du

111

122 135

144 156

Information access and Dissemination 2 Bayesian Method Based Trusted Overlay for Information Retrieval over Networks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Shunying L¨ u, Wei Wang, and Yan Zhang Managing a Geographic Database from Mobile Devices Through OGC Web Services . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Nieves R. Brisaboa, Miguel R. Luaces, Jose R. Parama, and Jose R. Viqueira Real-Time Creation Method of Personalized Mobile Web Contents for Ubiquitous Contents Access . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . SeungHyun Han, DongYeop Ryu, and YoungHwan Lim

168

174

180

Stream Data Management The Golden Mean Operator Scheduling Strategy in Data Stream Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Huafeng Deng, Yunsheng Liu, and Yingyuan Xiao

186

Continuous Skyline Tracking on Update Data Streams . . . . . . . . . . . . . . . . Li Tian, Le Wang, AiPing Li, Peng Zou, and Yan Jia

192

A Hybrid Algorithm for Web Document Clustering Based on Frequent Term Sets and k-Means . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Le Wang, Li Tian, Yan Jia, and Weihong Han

198

Table of Contents

XIX

Sensor, P2P, and Grid Networks 2 OMSI-Tree: Power-Awareness Query Processing over Sensor Networks by Removing Overlapping Regions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Wei Zha, Sang-Hun Eo, Byeong-Seob You, Dong-Wook Lee, and Hae-Young Bae On Studying Front-Peer Attack-Resistant Trust and Reputation Mechanisms Based on Enhanced Spreading Activation Model in P2P Environments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Yufeng Wang, Yoshiaki Hori, and Kouichi Sakurai A Load Balancing Method Using Ring Network in the Grid Database . . . Yong-Il Jang, Ho-Seok Kim, Sook-Kyung Cho, Young-Hwan Oh, and Hae-Young Bae Design and Implementation of a System for Environmental Monitoring Sensor Network . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Yang Koo Lee, Young Jin Jung, and Keun Ho Ryu Policy Based Scheduling for Resource Allocation on Grid . . . . . . . . . . . . . . Sung-hoon Cho, Moo-hun Lee, Jang-uk In, Bong-hoi Kim, and Eui-in Choi

204

211 217

223 229

Potpourri Characterizing DSS Workloads from the Processor Perspective . . . . . . . . . Dawei Liu, Shan Wang, Biao Qin, and Weiwei Gong

235

Exploiting Connection Relation to Compress Data Graph . . . . . . . . . . . . . Jun Zhang, Zhaohui Peng, Shan Wang, and Jiang Zhan

241

Indexing the Current Positions of Moving Objects on Road Networks . . . Kyoung Soo Bok, Ho Won Yoon, Dong Min Seo, Su Min Jang, Myoung Ho Kim, and Jae Soo Yoo

247

International Workshop on Emerging Trends of Web Technologies and Applications (WebETrends 2007) Keynote Talk DBMSs with Native XML Support: Towards Faster, Richer, and Smarter Data Management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Min Wang

253

Session 1 A Personalized Re-ranking Algorithm Based on Relevance Feedback . . . . Bihong Gong, Bo Peng, and Xiaoming Li

255

XX

Table of Contents

An Investigation and Conceptual Models of Podcast Marketing . . . . . . . . Shuchih Ernest Chang and Muharrem Cevher

264

A User Study on the Adoption of Location Based Services . . . . . . . . . . . . Shuchih Ernest Chang, Ying-Jiun Hsieh, Tzong-Ru Lee, Chun-Kuei Liao, and Shiau-Ting Wang

276

Email Community Detection Using Artificial Ant Colony Clustering . . . . Yan Liu, QingXian Wang, Qiang Wang, Qing Yao, and Yao Liu

287

EviRank: An Evidence Based Content Trust Model for Web Spam Detection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Wei Wang, Guosun Zeng, Mingjun Sun, Huanan Gu, and Quan Zhang A Novel Factoid Ranking Model for Information Retrieval . . . . . . . . . . . . . Youcong Ni and Wei Wang

299

308

Session 2 Dynamic Composition of Web Service Based on Coordination Model . . . Limin Shen, Feng Li, Shangping Ren, and Yunfeng Mu

317

An Information Retrieval Method Based on Knowledge Reasoning . . . . . Mei Xiang, Chen Junliang, Meng Xiangwu, and Xu Meng

328

Research on Personalized Recommendation Algorithm in E-Supermarket System . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Xiong Feng and Qi Luo

340

XML Normal Forms Based on Constraint-Tree-Based Functional Dependencies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Teng Lv and Ping Yan

348

Untyped XQuery Canonization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Nicolas Travers, Tuyˆet Trˆ am Dang Ngoc, and Tianxiao Liu

358

Web Search Tailored Ontology Evaluation Framework . . . . . . . . . . . . . . . . Darijus Strasunskas and Stein L. Tomassen

372

International Workshop on Process Aware Information Systems (PAIS 2007) Full Paper An Overview of the Business Process Maturity Model (BPMM) . . . . . . . . Jihyun Lee, Danhyung Lee, and Sungwon Kang

384

Table of Contents

XXI

Process Mining: Extending “α-Algorithm” to Mine Duplicate Tasks in Process Logs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Jiafei Li, Dayou Liu, and Bo Yang

396

A Distributed Genetic Algorithm for Optimizing the Quality of Grid Workflow . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Hongxia Tong, Jian Cao, and Shensheng Zhang

408

Safety Analysis and Performance Evaluation of Time WF-nets . . . . . . . . . Wei Song, Wanchun Dou, Jinjun Chen, and Shaokun Fan

420

Dual Workflow Nets: Mixed Control/Data-Flow Representation for Workflow Modeling and Verification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Shaokun Fan, Wanchun Dou, and Jinjun Chen

433

Bridging Organizational Structure and Information System Architecture Through Process . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Vladim´ır Modr´ ak

445

Relation-Driven Business Process-Oriented Service Discovery . . . . . . . . . . Yu Dai, Lei Yang, and Bin Zhang

456

Workflow Message Queue’s Performance Effect Measurements on an EJB-Based Workflow Management System . . . . . . . . . . . . . . . . . . . . . . . . . . Hyungjin Ahn, Kiwon Lee, Taewook Kim, Haksung Kim, and Ilkyeun Ra

467

Short Paper RFID Application Model and Performance for Postal Logistics . . . . . . . . . Jeong-Hyun Park and Boo-Hyung Lee

479

An Organization and Task Based Access Control Model for Workflow System . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Baoyi Wang and Shaomin Zhang

485

A Method of Web Services Composition Based on Service Alliance . . . . . Chunming Gao, Liping Wan, and Huowang Chen

491

Toward a Lightweight Process-Aware Middleware . . . . . . . . . . . . . . . . . . . . Weihai Yu

497

Automatic Generation of Web Service Workflow Using a Probability Based Process-Semantic Repository . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Dong Yuan, Miao Du, Haiyang Wang, and Lizhen Cui

504

A Three-Dimensional Customer Classification Model Based on Knowledge Discovery and Empirical Study . . . . . . . . . . . . . . . . . . . . . . . . . . Guoling Lao and Zhaohui Zhang

510

XXII

Table of Contents

QoS-Driven Global Optimization of Services Selection Supporting Services Flow Re-planning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Chunming Gao, Meiling Cai, and Huowang Chen

516

SOA-Based Collaborative Modeling Method for Cross-Organizational Business Process Integration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Hongjun Sun, Shuangxi Huang, and Yushun Fan

522

Model Checking for BPEL4WS with Time . . . . . . . . . . . . . . . . . . . . . . . . . . Chunming Gao, Jin Li, Zhoujun Li, and Huowang Chen

528

A Version Management of Business Process Models in BPMS . . . . . . . . . . Hyerim Bae, Eunmi Cho, and Joonsoo Bae

534

Research on Architecture and Key Technology for Service-Oriented Workflow Performance Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Bo Liu and Yushun Fan

540

International Workshop on Application and Security Service in Web and Pervasive Environments (ASWAN 2007) WPE Models and Applications The Study on Internet-Based Face Recognition System Using Principal Component Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Myung-A Kang and Jong-Min Kim

546

Semantic Representation of RTBAC: Relationship-Based Access Control Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Song-hwa Chae and Wonil Kim

554

A Code-Based Multi-match Packet Classification with TCAM . . . . . . . . . Zhiwen Zhang and Mingtian Zhou

564

Security and Services of WPE Home Network Device Authentication: Device Authentication Framework and Device Certificate Profile . . . . . . . . . . . . . . . . . . . . . . . . . . . Yun-kyung Lee, Deok Gyu Lee, Jong-wook Han, and Kyo-il Chung

573

P-IDC: Information Security and Consideration in Building Internet Data Centers for Pervasive Environment . . . . . . . . . . . . . . . . . . . . . . . . . . . . Jae-min Yang, Jong-geun Kim, and Jong-in Im

583

Improvement of an Authenticated Key Agreement Protocol . . . . . . . . . . . Yongping Zhang, Wei Wei, and Tianjie Cao

593

Table of Contents

A Mechanism for Securing Digital Evidences in Pervasive Environments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Jae-Hyeok Jang, Myung-Chan Park, Young-Shin Park, Byoung-Soo Koh, and Young-Rak Choi A Secure Chaotic Hash-Based Biometric Remote User Authentication Scheme Using Mobile Devices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Eun-Jun Yoon and Kee-Young Yoo

XXIII

602

612

WSN/RFID/Web Services Adapting Web Services Security Standards for Mobile and Wireless Environments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Nelly A. Delessy and Eduardo B. Fernandez Coexistence Proof Using Chain of Timestamps for Multiple RFID Tags . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Chih-Chung Lin, Yuan-Cheng Lai, J.D. Tygar, Chuan-Kai Yang, and Chi-Lung Chiang

624

634

A Design of Authentication Protocol for Multi-key RFID Tag . . . . . . . . . . Jiyeon Kim, Jongjin Jung, Hoon Ko, Boyeon Kim, Susan Joe, Yongjun Lee, Yunseok Chang, and Kyoonha Lee

644

An Efficient Fragile Watermarking for Web Pages Tamper-Proof . . . . . . . Chia-Chi Wu, Chin-Chen Chang, and Shang-Ru Yang

654

Classification of Key Management Schemes for Wireless Sensor Networks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Hwaseong Lee, Yong Ho Kim, Dong Hoon Lee, and Jongin Lim

664

Data Management and Access Control for WPE An Efficient Algorithm for Proportionally Fault-Tolerant Data Mining . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Tianding Chen

674

SOGA: Fine-Grained Authorization for Self-Organizing Grid . . . . . . . . . . Ming Guo, Yong Zhu, Yuheng Hu, and Weishuai Yang

684

Permission-Centric Hybrid Access Control . . . . . . . . . . . . . . . . . . . . . . . . . . Sejong Oh

694

Author Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

705

Cost Framework for a Heterogeneous Distributed Semi-structured Environment Tianxiao Liu1 , Tuyêt Trâm Dang Ngoc2 , and Dominique Laurent3 1

ETIS Laboratory - University of Cergy-Pontoise & XCalia S.A, France [email protected], [email protected] 2 ETIS Laboratory-University of Cergy-Pontoise, France [email protected] 3 ETIS Laboratory-University of Cergy-Pontoise, France [email protected]

Abstract. This paper proposes a generic cost framework for query optimization in an XML-based mediation system called XLive, which integrates distributed, heterogeneous and autonomous data sources. Our approach relies on cost annotation on an XQuery logical representation called Tree Graph View (TGV). A generic cost communication language is used to give an XML-based uniform format for cost communication within the XLive system. This cost framework is suitable for various search strategies to choose the best execution plan for the sake of minimizing the execution cost. Keywords: Mediation system, query optimization, cost model, Tree Graph View, cost annotation.

1

Introduction

The architecture of mediation system has been proposed in [Wie92] for solving the problem of integration of heterogeneous data sources. In such an architecture, users send queries to the mediator, and the mediator processes these queries with the help of wrappers associated to data sources. Currently, the semi-structured data model represented by XML format is considered as a standard data exchange model. XLive [NJT05], mediation system based on XML standard, has a mediator which can accept queries in the form of XQuery [W3C05] and return answers. The wrappers give the mediator an XML-based uniform access to heterogeneous data sources. For a given user query, the mediator can generate various execution plans (referred to as "plan" in the remainder of this paper) to execute it, and these plans can differ widely in execution cost (execution time, price of costly connections, communication cost, etc. An optimization procedure is thus necessary to determine the most efficient plan with the least execution cost. However, how to choose the best plan based on the cost is still an open issue. In relational or object-oriented databases, the cost of a plan can be estimated by using a cost model. This estimation is processed with database statistics and cost formulas K.C. Chang et al. (Eds.): APWeb/WAIM 2007 Ws, LNCS 4537, pp. 1–11, 2007. c Springer-Verlag Berlin Heidelberg 2007 

2

T. Liu, T.T. Dang Ngoc, and D. Laurent

for each operator appearing in the plan. But in a heterogeneous and distributed environment, the cost estimation is much more difficult, due to the lack of underlying databases statistics and cost formulas. Various solutions for processing the overall cost estimation at the mediator level have been proposed. In [DKS92], a calibration procedure is described to estimate the coefficients of a generic cost model, which can be specialized for a class of systems. This solution is extended for object database systems in [GGT96][GST96]. The approach proposed in [ACP96] records cost information (results) for every query executed and reuses that information for the subsequent queries. [NGT98] uses a cost-based optimization approach which combines a generic cost model with specific cost information exported by wrappers. However, none of these solutions has addressed the problem of overall cost estimation in a semi-structured environment integrating heterogeneous data sources. In this paper, we propose a generic cost framework for an XML-based mediation system, which integrates distributed, heterogeneous and autonomous data sources. This framework allows to take into account various cost models for different types of data sources with diverse autonomy degrees. These cost models are stored as annotations in an XQuery logical representation called Tree Graph View (TGV) [DNGT04][TDNL06]. Moreover, cost models are exchanged between different components of the XLive system. We apply our cost framework to compare the execution cost of candidate plans in order to choose the best one. First, we summarize different cost models for different types of data sources (relational, object oriented and semi-structured) and different autonomy degrees of these sources (proprietary, non-proprietary and autonomous). The overall cost estimation relies on the cost annotation stored in corresponding components TGV. This cost annotation derives from a generic annotation model which can annotate any component (i.e. one or a group of operators) of a TGV. Second, in order to perform the cost communication within the XLive system during query optimization, we define an XML-based language to express the cost information in a uniform, complete and generic manner. This language, which is generic enough to take into account any type of cost information, is the standard format for the exchange of cost information in XLive. The paper is organized as follows: In Section 2, we introduce XLive system with its TGV modeling of XQuery and we motivate our approach to cost-based optimization. In Section 3, we describe the summarized cost models and show how to represent and exchange these cost models using our XML-based generic language. Section 4 provides the description of TGV cost annotation and the procedure for the overall cost estimation at the mediator level. We conclude and give directions for future work in Section 5.

2

Background

XQuery Processing in XLive. A user’s XQuery submitted to the XLive mediator is first transformed into a canonical form. Then the canonized XQuery is modeled in an internal structure called TGV. We annotate the TGV with

Cost Framework

3

information on evaluation, such as the data source locations, cost models, sources functional capabilities of sources, etc. The optimal annotated TGV is then selected based on a cost-based optimization strategy. In this optimization procedure, TGV is processed as the logical execution plan and the cost estimation of TGV is performed with cooperation between different components of XLive. This optimal TGV is then transformed into an execution plan using a physical algebra. To this end, we have chosen the XAlgebra [DNG03] that is an extension to XML of the relational algebra. Finally, the physical execution plan is evaluated and an XML result is produced, Fig.1 depicts the different steps of this processing.

. . . . . .

Users

Users

Response

Query

Query Result (XML)

XQuery

Mediator Evaluation

Canonization

Equivalent rules

Search Strategy

XAlgebra Canonized XQuery

Mediator

Cost-based Optimization

Transformation

Tree Graph Views(TGV)

Mediator cost information

Mediator Operators

Annotated TGV

Modeling

Mediator Information Repository

Wrappers Operators Annotation

Static wrappers cost information

Wrapper Information Repository

Dynamic Wrappers Cost information Wrapper

Wrapper Sources

Wrapper

Relational data bases

Wrapper

XML data sources

Web services

. . . . . .

Fig. 1. Cost-based optimization in processing of XQuery in the XLive system

Tree Graph View. TGV is a logical structure model implemented in the XLive mediator for XQuery processing, which can be manipulated, optimized and evaluated [TDNL06]. TGV takes into account the whole functionality of XQuery (collection, XPath, predicate, aggregate, conditional part, etc.) and uses an intuitive representation that provides a global view of the request in a mediation context. Each element in the TGV model has been defined formally using Abstract Data Type in [Tra06] and has a graphical representation. In Fig. 2 (a), we give an example of XQuery which declares two FOR clauses ($a and $b), a join constraint between authors and a contains function, then a return clause projects the title value of the first variable. This query is represented by a TGV in Fig. 2 (b). We can distinguish the two domain variables $a and $b of the XQuery, defining each nodes corresponding to the given XPaths. A join hyperlink links the two author nodes with an equality annotation. The contains function is linked to the $b "author" node, and a projection hyperlink links the node title to the ReturnTreePattern in projection purposes.

4

T. Liu, T.T. Dang Ngoc, and D. Laurent

for $a in col("catalogs")/catalog/book for $b in col("reviews")/reviews/review where $a/author = $b/author and contains($b/author,"Hobb")

"catalogs"

"reviews"

catalog

reviews

$b

$a

book

title

review

author

=

author contains("Hobb")

return {$a//title}

(a) An XQuery query

books

(b) TGV representation

Fig. 2. An example of XQuery and its TGV representation

TGV Generic Annotation. The motivation to annotate a TGV is to allow annotating subsets of elements of a TGV model with various information. Precisely, for each arbitrary component (i.e. one or a group of operators of TGV), we add some additional information such as cost information, system performance information, source localization, etc. Our annotation model is generic and allows annotation of any type of information. The set of annotation based on the same annotation type is called an annotated view. There can be several annotated views for the same TGV, for example, time-cost annotated view, algorithm annotated view, sources-localization annotated view, etc.

3 3.1

Heterogeneous Cost Models and Cost Communication Within XLive Cost Models for Heterogeneous Autonomous Data Sources

Cost Models Summary. We summarize different existing cost models for various types of data sources in Fig. 3. This summary is not only based on types of data sources but also on autonomy degrees of these sources. In addition, this summary gives some relations between different works on cost-based query optimization. The cost models with the name "operation" contain accurate cost formulas for calculating the execution cost of operators appearing in the plan. Generally, cost information such as source statistics is necessary for these cost models, because these statistics are used to derive the value of coefficients in cost formulas. It is often data sources implementers who are able to give accurate cost formulas with indispensable sources statistics. When the data s ources are autonomous, cost formulas and source statistics are unavailable. For obtaining cost models we need some special methods that vary with the autonomy degree of data sources. For example, the method by Calibration [DKS92] estimates the coefficients of a generic cost model for each type of relational data sources. This calibration needs to know access methods

Cost Framework

Cost models based on operation implemetation

Calibration [DKS92] Operation [GP89] [ML86] [SA82]

Specific methods for obtaining cost

Generic cost models Calibration procedure unavailable

5

Sampling [ZL98]

Historical cost [ACP96]

Adaptive [Zhu95]

Extended

Applied Operation [CD92] [BMG93] [DOA+94]

Calibration [GST96]

Flora [Flo96] [Gru96]

Wrappers [HKWY97] [ROH99]

Applied Operation [AAN01] [MW99]

Path [GGT96]

Proprietary data sources Relational data sources

XQuery Self-Learning [ZHJGML05]

Hybrid cost model [NGT98]

Heterogeneous autonomous data sources Object-oriented data sources

Semi-structured data sources

Fig. 3. Cost models for heterogeneous sources

used by the source. This method is extended to object-oriented databases by [GST96]. If this calibration procedure can not be processed due to data source constraints, a sampling method proposed in [ZL98] can derive a cost model for each type of query. The query classification in [ZL98] is based on a set of common rules adopted by many DBMSs. When no implementation algorithm and cost information are available, we can use the method described in [ACP96], in which cost estimation of new queries is based on the history of queries evaluated so far. Generic Cost Model. Here, we show how to reuse the summary in Fig. 3 to define our generic cost model used for XQuery optimization in the XLive system. First, a cost model is generally designed for some type of data source (but there are also some methods that can be used for different types of sources, for example, the method by history [ACP96]). Second, this cost model can contain some accurate cost formulas with coefficients’ value derived from data sources statistics, or a specific method for deriving the cost formulas. This cost model may also have only a constant value for giving directly the execution cost of operators. The possible attributes of our generic cost model are described in Table 1. This descriptive definition of cost model is used for TGV cost annotation for the purpose of overall cost estimation in the mediator level (ref. Section 4). For a cost model, all attributes are optional by reason of generality. We apply the principle as accurate as possible. For example, the method by calibration can normally provide more accurate cost models than the method based on historical costs, but it has a lower accuracy level than cost models based on operation implementation. That means if the cost models based on operations implementation are available, we use neither the method by calibration nor history.

6

T. Liu, T.T. Dang Ngoc, and D. Laurent Table 1. Definition of generic cost model

3.2

Generic Language for Cost Communication (GLCC)

XML-Based Generic Language. To perform cost communication within our XLive system, we define a language to express the cost information in a uniform, complete and generic manner. This language fits to our XML environment, to avoid costly format converting. It considers every cost model type and allows wrappers to export their specific cost information. In our XLive context, this language is generic enough to express cost information of different parts of a TGV and is capable to express cost for various optimization goals, for example, response time, price, energy consummation, etc. Our language extends the MathML language [W3C03], which allows us to define all mathematical functions in XML form. MathML fits to cost communication within XLive due to its semi-structured nature. We use the Content Markup in MathML to provide explicit encoding for cost formulas. We just add some rules to MathML to define the grammar of our language. Furthermore, this grammar is extensible so that users can always define its own tags for any type of cost. Cost formulas are represented in the form of equations set. Each equation corresponds to a cost function that may be defined by the source or by the mediator. Each component of TGV is annotated with an equation set in which the number of equations is undefined. One function in a set may use variables defined in other sets. We define some rules to ensure the consistency of the equations system. First, every variable should have somewhere a definition. Second, by

Cost Framework

7

Mediator TGV cost computation

Provides Cost Information

Historical cost Adjustment Historical of cost models Records

Wrapper Information Repository

Records

Cost models

Parser

Adjustment of cost models Cost information

Extract Data Source

Wrapper Operators Evaluation

An example of cost model and its representation on GLCC Cost_Re = Cost_Restriction + Cost_Projection <eq/> CostRe CostRestriction CostProjection ......

Information transfered using GLCC.

Fig. 4. Dynamic cost evaluation with GLCC in XLive system

reason of generality, there are no predefined variable names. For example, in the grammar, we do not define a name "time" for a cost variable because the cost metric can be a price unit. It is the user of the language who gives the specific significant names to variables. This gives a much more generic cost definition model compared to the language defined in [NGT98]. Dynamic Cost Evaluation. Fig. 4 gives a simple example for the expression of a cost model and shows the role of our language in cost communication. After extracting cost information from data source, the wrapper exports that information using our language to the parser, which derives cost models that will be stored in the wrapper information repository. When the mediator needs to compute the execution cost of a plan (TGV), the wrapper information repository provides necessary cost information for operators executed on wrappers. We have a cache for storing historical execution cost of queries evaluated, which can be used to adjust the exported cost information from the wrapper. All these communications are processed in the form of our language. Our language completes the interface between different components of XLive.

4 4.1

Overall Cost Estimation TGV Cost Annotation

As mentioned in Section 2, the TGV is the logical execution plan of XQuery within the query processing in XLive. The purpose of our query optimization is to find the optimal TGV with the least execution cost. For estimating the overall cost of a TGV, we annotate different components (one or a group of operators) of

8

T. Liu, T.T. Dang Ngoc, and D. Laurent

TGV. For an operator or a group of operators appearing in a TGV, the following cost information can be annotated: – Localization: The operator(s) can be executed on the mediator or on the wrappers (data sources). – Cost Model: Used to calculate the execution cost of the component. – Other information: Contains supplementary information that is useful for cost estimation. For example, several operators’ (such as join operator) implementation allows parallel execution between its related operators. (1)

(6) "catalogs"

"reviews"

catalog

reviews

(2) (3)

$b

$a

book

review

(4) title

author

= (7)

author contains("Hobb")

(5) (8)

books

(9)

card: cardinality

sel: selectivity

restr: restriction

proj: projection

Fig. 5. An example for TGV cost annotation

Fig. 5 gives an example for TGV cost annotation. In this example, different components of the TGV introduced in Fig. 2 (Ref. Section 2) are annotated. We can see for the operators executed on Source1(S1), we have only the historical cost to use for estimate the total execution cost of all the these operators; in contrast, for each operator executed on Source2(S2), we have a cost model for estimating its execution cost. For the join operator(numbered (7)) executed on the mediator, the operators linked to it can be executed in parallel. 4.2

Overall Cost Estimation

Cost Annotation Tree (CAT). We have seen how to annotate a TGV with cost information. Now we are concentrated on how to use this cost annotation for the overall cost estimation of a TGV. As illustrated in Fig. 5, the cost of an annotated component of TGV generally depends on the cost of other components. For example, for the cost formula annotated in (6), we see that it depends on the cost of (2), (3), (4) and (5). From the cost formulas annotated for each component of TGV, we obtain a Cost Annotation Tree (CAT). In a CAT, each node represents a component of TGV annotated by cost information and this CAT describes the hierarchical relations between these different components. Fig. 6 (a) illustrates the CAT of the TGV annotated in Fig. 5.

Cost Framework

9

1

2

1

6

3

7

4

8

5

9

1 associateCost (node) { 2 node.analyzeCostModel ( ); 3 if (node.hasSpecialMethod( )) { 4 node.callAPI( ); 5 } 6 for (each child of node) { 7 associateCost(child); 8 } 9 node.configCostFormula( ); 10 node.calculateCost( ); 11 }

Node that needs to call APIs for obtaining the necessary coefficients’ value

(a) Cost Annotation Tree (CAT)

(b) Overall cost estimation algorithm

Fig. 6. Cost Annotation Tree and the algorithm for overall cost estimation

Overall Cost Estimation Algorithm. We now show how to use the CAT of a TGV to perform the overall cost estimation. We use the recursive breadthfirst search algorithm of a tree for performing cost estimation of each node. For each node of CAT, we define a procedure called associateCost (Fig. 6 (b)) for operating the cost annotation of a node. This procedure first analyzes the cost annotation of the node and derives its cost model (line 2); If a specific cost method is found, it calls an API implemented by XLive for obtaining the necessary values of coefficients or cost formulas for computing the cost (line 3-5); if the cost of this node depends on the cost of its child nodes, it executes recursively the associateCost procedure on its child nodes (line 6-8). When these 3 steps are terminated, a procedure configCostFormula completes the cost formulas with obtained values of coefficients (line 9) and execution cost of this node will be calculated (line 10). By using this algorithm, we can obtain the overall cost of a TGV, which is the cost of the root of CAT. 4.3

Application: Plan Comparison and Generation

It has been shown in [TDNL06] that for processing a given XQuery, a number of candidate plans (i.e. TGV) can be generated using transformation rules that operate on TGVs. These rules have been defined for modifying the TGV without changing the result. The execution cost of a TGV can be computed by using our generic cost framework and thus we can compare the costs of these plans to choose the best one to execute the query. However, as the number of rules is huge, this implies an exponential blow-up of the candidate plans. It is impossible to calculate the cost of all these candidate plans, because the cost computation and the subsequent comparisons will be even more costly than the execution of the plan. Thus, we need a search strategy to reduce the size of the search space containing candidate execution

10

T. Liu, T.T. Dang Ngoc, and D. Laurent

plans. We note in this respect that our cost framework is generic enough to be applied to various search strategies such as exhaustive, iterative, simulated annealing, genetic, etc.

5

Conclusion

In this paper, we described our cost framework for the overall cost estimation of candidate execution plans in an XML-based mediation system. The closest related work is DISCO system [NGT98], which defines a generic cost model for an object-based mediation system. Compared to DISCO work and other mediation systems, we have the following contributions: First, to our knowledge, our cost framework is the first approach proposed for addressing the costing problem in XML-based mediation systems. Second, our cost communication language is completely generic to express any type of cost, which is an improvement compared to the language proposed in DISCO. Third, our cost framework is generic enough to fit to overall cost computation within various mediation systems. As futur work, we plan to define a generic cost model for XML sources with cost formulas that can compute the cost with given parameters that are components in TGV. This cost model would be generic for all types of XML sources. We will also concentrate on the design of an efficient search strategy that will be used in our cost-based optimization procedure.

Acknowledgment This work is supported by Xcalia S.A. (France) and by ANR PADAWAN project.

References [AAN01]

[ACP96]

[BMG93] [CD92] [DKS92] [DNG03] [DNGT04] [DOA+94]

Aboulnaga, A., Alameldeen, A., Naughton, J.: Estimating the Selectivity of XML Path Expressions for Internet Scale Applications. In: VLDB (2001) Adali, S., Candan, K., Papakonstantinou, Y.: Query Caching and Optimization in Distributed Mediator Systems. In: ACM SIGMOD (1996) Blakeley, J.A., McKenna, W.J., Graefe, G.: Experiences Building the Open OODB Query Optimizer. In: ACM SIGMOD (1993) Cluet, S., Delobel, C.: A General Framework for the Optimization of Object-Oriented Queries. In: ACM SIGMOD (1992) Du, W., Krishnamurthy, R., Shan, M.C.: Query Optimization in a Heterogeneous DBMS. In: VLDB (1992) Dang-Ngoc, T.T., Gardarin, G.: Federating Heterogeneous Data Sources with XML. In: Proc. of IASTED IKS Conf. (2003) Dang-Ngoc, T.T., Gardarin, G., Travers, N.: Tree Graph View: On Efficient Evaluation of XQuery in an XML Mediator. In: BDA (2004) Dogac, A., Ozkan, C., Arpinar, B., Okay, T., Evrendilek, C.: Advances in Object-Oriented Database Systems. Springer-Verlag (1994)

Cost Framework [Flo96] [GGT96]

[GM93] [GP89] [Gru96] [GST96] [HKWY97] [ML86] [MW99] [NGT98] [NJT05]

[ROH99]

[RS97] [SA82] [TDNL06]

[Tra06] [W3C03] [W3C05] [Wie92] [ZHJGML05] [Zhu95]

[ZL98]

11

Florescu, D.: Espace de Recherche pour l’Optimisation de Requêtes Objet PhD thesis, University of Paris IV (1996) Gardarin, G., Gruser, J.R., Tang, Z.H.: Cost-based Selection of Path Expression Algorithms in Object-Oriented Databases. In: VLDB (1996) Graefe, G., McKenna, W.J.: The Volcano Optimizer Generator: Extensibility and Efficient Search. In: ICDE (1993) Gardy, D., Puech, C.: On the Effects of Join Operations on Relation Sizes. ACM Transactions on Database Systems (TODS) (1989) Gruser, J.R.: Modéle de Coût pour l’Optimisation de Requêtes Objet. PhD thesis, University of Paris IV (1996) Gardarin, G., Sha, F., Tang, Z.H.: Calibrating the Query Optimizer Cost Model of IRO-DB. In: VLDB (1996) Haas, L.M., Kossmann, D., Wimmers, E.L., Yang, J.: Optimization Queries Across Diverse Data Sources. In: VLDB (1997) Mackert, L.F., Lohman, G.M.: R* Optimizer Validation and Performance Evaluation for Local Queries. In: ACM SIGMOD (1986) McHugh, J., Widom, J.: Query Optimization for Semistructured Data. Technical report, Stanford University Database Group (1999) Naacke, H., Gardarin, G., Tomasic, A.: Leveraging Mediator Cost Models with Heterogeneous Data Sources. In: ICDE (1998) Dang Ngoc, T.T., Jamard, C., Travers, N.: XLive: An XML Light Integration Virtual Engine. In: Bases de Données Avancées (BDA) (2005) Roth, M.T., Ozcan, F., Haas, L.: Cost Models DO Matter: Providing Cost Information for Diverse Data Sources in a Federated System. In: VLDB (1999) Tork Roth, M., Schwarz, P.M.: Don’t Scrap It, Wrap it! A Wrapper Architecture for Legacy Data Sources. In: VLDB (1997) Selinger, P.G., Adiba, M.E.: Access path selection in distributed database management systems. In: ICOD (1982) Travers, N., Dang-Ngoc, T.T., Liu, T.: TGV: An Efficient Model for XQuery Evaluation within an Interoperable System. Int. Journal of Interoperability in Business Information Systems (IBIS), vol. 3 (2006) Travers, N.: Optimization Extensible dans un Mèdiateur de Donnèes XML, PhD thesis, University of Versailles (2006) W3C. Mathematical Markup Language (Mathml TM) Version 2.0 (2003) W3C. An XML Query Language (XQuery 1.0) (2005) Wiederhold, G.: Mediators in the Architecture of Future Information Systems. Computer 25(3), 38–49 (1992) Zhang, N., Haas, P.J., Josifovski, V., Zhang, C., Lohman, G.M.: Statistical Learning Techniques for Costing XML Queries. In: VLDB (2005) Zhu, Q.: Estimating Local Cost Parameters for Global Query Optimization in a Multidatabase System. PhD thesis, University of Waterloo (1995) Zhu, Q., Larson, P.A.: Solving Local Cost Estimation Problem for Global Query Optimization in Multidatabase Systems. Distributed and Parallel Databases (1998)

Cost-Based Vertical Fragmentation for XML Sven Hartmann, Hui Ma, and Klaus-Dieter Schewe Massey University, Department of Information Systems & Information Science Research Centre, Private Bag 11 222, Palmerston North, New Zealand {s.hartmann,h.ma,k.d.schewe}@massey.ac.nz

Abstract. The Extensible Markup Language (XML) has attracted much attention as a data model for data exchange, data integration and rich data representation. A challenging question is how to manage native XML data in distributed databases. This leads to the problem of how to obtain a suitable distribution design for XML documents. In this paper we present a design approach for vertical fragmentation to minimise total query costs. Our approach is based on a cost model that takes the complex structure of queries on XML data into account. We show that system performance can be improved after vertical fragmentation using our approach, which is based on user access patterns.

1

Introduction

Vertical fragmentation is an important database distribution design technique for improving system performance. It has been widely studied in the context of the relational model, cf. [6, 10, 18, 19, 20, 21, 23] and object oriented data model, cf. [3, 7, 11]. With the emergence of XML as a standard format for data exchange, data integration and rich data representation, distribution design for XML becomes a highly relevant topic, in particular as data shared over the web is naturally distributed to meet the needs of the users who are physically distributed. Due to the particularities of the XML data model, the adaption of distribution design principles for relational or object-oriented data to XML poses a real challenge for database research. In [12], horizontal, vertical, and split fragmentation techniques for XML are studied. In [13], a horizontal fragmentation algorithm for XML is proposed. In [5], a fragmentation method for XML is proposed together with an allocation model for distributed XML fragments. For that, local index structures are presented that allow efficient storage of global context for local fragments, facilitate the local execution of queries, and support the reconstruction of fragments distributed over multiple sites. In [2], horizontal, vertical, and hybrid fragmentation techniques for XML are surveyed. In addition, correctness rules evaluating a fragmentation schema are presented and experimental results are given to emphasise the beneficial effects of fragmentation. To the best of our knowledge no work so far has discussed how to perform fragmentation in the context of XML procedurally using an top-down design approach based on user queries as input information. In the paper at hand, we K.C. Chang et al. (Eds.): APWeb/WAIM 2007 Ws, LNCS 4537, pp. 12–24, 2007. c Springer-Verlag Berlin Heidelberg 2007 

Cost-Based Vertical Fragmentation for XML

13

will study this problem for the vertical fragmentation of XML data. Most vertical fragmentation algorithms for relational data presented in the literature are affinity-based, that is, they use attribute affinities as input. Affinity-based fragmentation approaches have also been adopted to the object oriented data model, using different kinds of affinities such method affinities, attribute affinities, or instance variable affinities. One disadvantage of affinity-based approaches is that they do not reflect local data requirements and therefore cannot improve the local availability of data and reduce data transportation between sites to improve system performance. To overcome this deficiency of affinity-based algorithms, a cost-based vertical fragmentation approach for relational data has recently been proposed in [14]. In this paper, we will discuss cost-based vertical fragmentation in the context of XML, thus extending earlier work presented in [16, 14]. In particular, we will outline a cost model that incorporates available information about user queries issued at various sites of the system. Using such a cost model it is possible to handle vertical fragmentation and fragment allocation simultaneously with low computational complexity and resulting high system performance. The remainder of this paper is organised as follows. In the remainder of this section we start with a brief review of the XML data model, discuss query algebra for XML, and give a definition of vertical fragmentation of XML data. In Section 2, we present a cost model for XML. In Section 3, we propose a costbased design approach for vertical fragmentation and illustrate it by an example. We conclude with a short summary in Section 4. XML as Native Data Type. With the emergence of XML, most providers of commercial database management systems made an effort to incorporate some support for XML data management into their products. In the beginning, this was typically achieved by shredding XML data to object-relational structures or by mapping it to existing data types such as LOBs (CLOBs, BLOBs). The limitations of this approach are well-known. Since then, much effort has been spent to optimise database management systems for the handling of XML data. Only recently, native XML support has been implemented, e.g., in the latest version of IBM DB2 [22] where XML can now be included into tables using the new XML data type. That is, tables may hold relational data and native XML data simultaneously. In SQL table declarations, it is now possible to specify that a column should be of type XML, cf. Example 1. Example 1. With the XML data type, we may for example define a table with the table schema Department = (Name : STRING, Homepage : STRING, Address : STRING, Lecturers: XML, Papers: XML). Internally, XML and relational data are stored in different formats, which match their corresponding data models. XML columns are stored on disk pages in tree structures matching the XML data model. Using the common XML data model, XML documents are regarded as data trees. XML documents are frequently described by XML schemas. Most database management systems support XSDs

14

S. Hartmann, H. Ma, and K.-D. Schewe lecturers

lecturer

ID

name

email

phone

STRING

STRING fname

lname

STRING

STRING

titles

homepage

areacode

INTEGER

number

INTEGER

title

STRING

Fig. 1. XML Tree for the Lecturer Data Type

(XML Schema Definitions) as XML schemas. XML schemas can be visualised as schema trees, too. This often helps imagining the nested structure of the corresponding XML documents. In our example, we might be given an XML schema for the Lecturers attribute with the schema tree in Figure 1. When declaring a column as XML, each cell in this column may hold an XML document. That is, the domain of the XML data type is actually complexvalued. For the sake of simplicity, we assume that the XML documents in such a column are described by an XML schema. In DB2, for example, XSDs (XML Schema Definitions) can be used for this purpose. To facilitate the adaption of the vertical fragmentation technique from the relational model to XML, it is useful to assume that each XML element e has a label  and a content type t. In particular, each column  of type XML is thus associated with the root element e of an XML schema with label  and content type t. For the expressiveness of different XML schema languages and content types of elements, we refer to [4]. In the paper at hand, we will also use the label-extended type system defined in [16] where content types are called representation types. Example 2. Consider Lecturers and Papers that have been declared as XML. From the XML schema we obtain their representation types which may look for examples as follows: tLecturers = lecturers: {lecturer: (id: INTEGER, name: (fname: STRING, lname: STRING, titles:{title: STRING}), homepage: STRING, email: STRING, phone: (areacode: INTEGER, number: INGEGER))} tPapers = papers: {paper: (no : STRING, title: STRING, taught : IDREF , description: STRING, points: STRING, campus: STRING, literature: [(article : STRING) ⊕ (book: STRING)])}

Cost-Based Vertical Fragmentation for XML

15

Query Algebra. Database management systems exploit query algebra for evaluating user queries. For that, user queries are transformed into query trees and further optimised during query optimisation. The query tree specifies which query plan is actually executed inside the database management system to compute the query result. Query algebras for XML have been studied extensively in the literature. Prominent examples include the XQuery algebra, the IBM and Lore algebra [9], and XAL [8]. In [1] query algebras for XML are surveyed, while in [24] query algebras for nested data structures are surveyed. In [16], a generic query algebra for complex value databases is presented. However, currently no complete theory of query optimisation based on structural recursion is known. For our purposes here, we restrict the discussion to a simpler query algebra that provides at least standard operations such as projection, selection, join, union, difference, and intersection. For projection, for example, we can use an operator πp1 ,...,pi (e) with path expressions p1 , . . . , pi . When applied to an XML element e the tree rooted at e will be reduced to the subtrees rooted at the subnodes selected by the specified path expression p1 , . . . , pk plus the paths from e to these subnodes. The label of e will be the same as before, while the content type can be updated to reflect the reductions. Each node in an XML tree can be reached from the root node by a unique path. The labels along this path give rise to a simple path expression 0 /1 / . . . /k . In DB2, for example, all labels that are used in XML documents are stored in a global string table [22]. As common, we use XPath expressions to access nodes in XML trees. To support query processing we need a path language that is expressive enough to be practical, yet sufficiently simple to be reasoned about efficiently. The computational tractability of query containment and query equivalence has been widely studied in the literature, cf. [17]. Here we use path expressions p which may include node labels, the context node (.), the child axis (/), the descendant-or-self axis (//), and the label wildcard (∗). That is, we use the XPath fragment XP [//,∗] for which query equivalence can be decided in polynomial time [17]. Example 3. The algebraic query expression πid, name/fname(lecturers) projects the lecturer elements to the lecturers’ id and first name. Vertical Fragmentation. Consider a table schema with a column  declared as XML, and let t = te be the representation type of the root element e of the corresponding XML schema in the label-extended type system [16]. Vertical fragmentation will replace the column  by new columns v1 , . . . , vi , where each vj is again declared as XML. Let tj denote the representation types of the root elements of the corresponding XML schemas for the new columns. Vertical fragmentation of e can be achieved by projection operations using path expressions that are evaluated from the root. The general rule of performing fragmentation is that it should be lossless, that is, there should not be any loss or addition of data after the fragmentation. In the RDM the correctness rules for fragmentation include three criteria, completeness,

16

S. Hartmann, H. Ma, and K.-D. Schewe

disjointness and reconstruction. We now adapt them to vertical fragmentation of XML documents. Completeness requires that each leaf node (that is, element node with pure text content or attribute node) together with its attached data item in an XML document must appear in at least one of the fragments. Disjointness requires that each leaf node together with its attached data item in an XML document occur no more than one of the fragments. Reconstruction requires that it is possible to reconstruct the original XML document by using join operations. In other words, one has to ensure that the fragmentation is reversible.

2

A Query Processing Cost Model

In practice there are usually several query trees that can be used to compute the result of a query: queries may be rewritten in various equivalent forms, and intermediate results may be executed at different sites. In this section we look at a cost model for the queries that can be used to evaluate and compare query trees. For that, we suggest to estimate the sizes of intermediate results for all intermediate nodes in the query tree. These sizes determine the costs for retrieving data for the next step, i.e., the operation associated with the predecessor in the query tree, and the costs for the transportation of data between nodes. Afterwards, we present a cost model for measuring system performance. Size Estimation. We first approach an estimation of the sizes intermediate query results. In order to do so, we first look at elements and attributes defined in the XML schema. Then, if r is the root element, the estimated size s(r) is the size estimated for the entire document rooted at r. Let si be the average size of values of base type bi . In particular, let s0 be the average size of values of base types ID and IDREF , that is, of identifiers. From these assumptions we can compute estimates for the average size of elements. Let att() denote the set of attributes defined on element . We can proceed inductively to define the size s(e): ⎧ si ⎪ ⎪ ⎪  ⎪ ⎪ s(a) ⎪ ⎪ ⎪ a∈att()=1  ⎪ ⎪ s(n) + a∈att() s(a) ⎪ ⎪ ⎪ ⎨q · s(e ) +  a∈att() s(a) s(e) = n  ⎪ i=1 s(ei ) + a∈att() s(a) ⎪ ⎪ n  ⎪ ⎪ ⎪ ⎪ i=1 pi · s(ei ) + a∈att() s(a) ⎪  ⎪ n ⎪ 1 ⎪ s(e ) i ⎪ i=1 ⎪ ⎩n r · s(e )

if if if if if if if if

e = bi e= e=n e = (e )∗ e = e1 , . . . , en e = e1 , . . . , en e = (a1 : e1 ) ⊕ · · · ⊕ (an : en ) e = {e } or e = [e ] or e = e 

where q is the average number of successor elements with e , pi  is the probability n that the successor of  is defined by ei . In particular, we have i=1 pi = 1.

Cost-Based Vertical Fragmentation for XML

17

se where s is the size of sp the successor node in the query tree, sp is the average size of an element reached by p , se is the average size of an element defined by e, and p is the probability that two element x1 , x2 matching p coincide on their projection   to e, i.e.,πee (x1 ) = πee (x2 ). s1 s2 – For a join node p the size is  · p ·  (s1 + s2 − s), where the si (i = 1, 2) are s1 s2 sizes of the successors in query tree, si (i = 1, 2) are the sizes of the elements, e1 , e2 , p is the probability that such two elements match, and s is the size of common leaf nodes e.

– The size of a projection node πp is (1 − p) · s ·

Query Processing Costs. Taking the cost model in [15] we now analyse the query costs in the case of vertical fragmentation. For the convenience of discussion we briefly present the cost model in the following. The major objective is to base the fragmentation decision on the efficiency of the most frequent queries. Fragmentation results in a set of fragments {F1 , . . . , Fn } of average sizes s1 , . . . , sn . If the network has a set of nodes N = N1 , . . . , Nk we have to allocate these fragments to one of the nodes, which gives rise to a mapping λ : {1, . . . , n} → {1, . . . , k}, which we call a location assignment . This decides the allocation of leaves of query trees, which are fragments. For each intermediate node v in each relevant query tree, we must also associate a node λ(v), i.e., λ(v) indicating the node in the network that the intermediate query result corresponding to v will be stored at. Given a location assignment λ we can compute the total costs of query processing. Let the set of queries be Qm = {Q1 , . . . , Qm }, each of which is executed with a frequency fj . The total costs of all the queries in Qm are the sum of the costs of each query multiplied by its frequency. The cost of each query are composed of two parts, the storage costs and transportation costs. Storage costs measure the costs of retrieving the data back from secondary storage, depend on the size of the intermediate results and on the assigned locations, which decide the storage cost factors. The transportation costs provide a measure for transporting between two nodes of the network, depend on the sizes of the involved sets and on the assigned locations, which decide the transport cost factor between every pair of sites. Costsλ (Qm ) = =

m  j=1 m  j=1

(storλ (Qj ) + transλ (Qj )) · fj   ( s(h) · dλ(h) + cλ(h )λ(h) · s(h )) · fj h

h

h

where h ranges over the nodes of the query tree for Qj , s(h) are the sizes of the involved sets, and di indicates the storage cost factor for node Ni (i = 1, . . . , k), h runs over the predecessors of h in the query tree, and cij is the transportation cost factor for data transport from node Ni to node Nj (i, j ∈ {1, . . . , k}).

18

3

S. Hartmann, H. Ma, and K.-D. Schewe

Cost-Based Approach for Vertical Fragmentation

The aim of vertical fragmentation is to improve the system performance. In this section we will adapt the cost-efficient vertical fragmentation approach in [14] to XML. We start with some terminology, then present a vertical fragmentation design methodology, and illustrate it with an example. Assume an XML document with a representation type te being accessed by a set of queries Qm = {Q1 , . . . , Qj , . . . Qm } with frequencies f1 , . . . , fm , respectively. To improve the system performance, element e is vertically fragmented into a set of fragments {eV 1 , . . . , eV u , . . . , eV ki }, each of which is allocated to one of the network nodes N1 , . . . , Nθ , . . . , Nk . Note that the maximum number of fragments is k, i.e.,ki ≤ k. We use λ(Qj ) to indicate the site that issues query Qj , and use elemj = {ei |fji = fj } to indicate the set of leaf elements that are accessed by Qj , with fji as the frequency of the query Qj accessing ei . Here, fji = fj if the element ei is accessed by Qj . Otherwise, fji = 0. The term request of an element at a site θ is used to indicate the sum of frequencies of all queries at the site accessing the element or attribute: m  requestθ (ei ) = fji . j=1,λ(Qj )=θ

With the size i of an element ei , we can calculate the need of an element or attribute as the total data volume involved to retrieve ai by all the queries:  needθ (ei ) = fji · i . j=1,λ(Qj )

Finally, we introduce a term pay to measure the costs of accessing an element once it is allocated to a network node. The pay of allocating an element ei to a site θ measures the costs of accessing element ei by all queries from the other sites θ , which is different from theta. It can be calculated using the following formula: k m   payθ (ei ) = fji · chh . θ  =1,θ=θ  j=1,λ(Qj )=θ

A Design Methodology for Vertical Fragmentation. Following [20] we assume a simple transaction model for database management systems where the system collects the information at the site of the query and executes the query there. Under this assumption we can evaluate the costs of allocating a single element and then make decisions by choosing a site that leads to the least query costs. Following the discussion of how fragmentation affects query costs [16], the allocation of fragments to sites according to the cost minimisation heuristics already determines the location assignment, provided that an optimal location assignment for the queries was given prior to the fragmentation. We take a two-step approach for vertical fragmentation of a table. For columns of some conventional (relational) data type we use the heuristic [14] to assign attributes to fragments, such that they share the primary key attributes. For

Cost-Based Vertical Fragmentation for XML

19

columns of XML type we adapt the heuristic approach in [14] to perform vertical fragmentation with the following steps. Note that, for the sake of simplicity, we do not consider replication at this stage. To record query information we use a Element Usage Frequency Matrix (EUFM) which records frequencies of queries, the subset of leaf nodes accessed by the queries and the sites that issue the queries. Each row in the EUFM represents one query Qj ; the head of column is the set of nodes that are covered by the representation type te , listed in some hierarchical order from top to bottom. When considering nodes we do not distinguish between attribute and element nodes, but record them in the same matrix. In addition, there are two further columns with one column indicating the site that issues the queries and the other indicating the frequency of the queries. The values on a column indicate the frequencies fji of the queries Qj that use the corresponding leaf node ei grouped by the site that issues the queries. Note that we treat any two queries issued at different sites as different queries, even if the queries themselves are the same. The EUFM is constructed according to the optimised queries to record all the element requirements returned by queries as well as all the elements used in some join predicates. If a query returns all the information of an element then every descendant element are accessed by the query. As a general pragmatic guideline we follow the recommended rule of thumb to consider the 20% most frequent queries, as these usually account for most of the data access [23]. This procedure can be described as follows and implemented as in Table 1. 1. Take the most frequently processed 20% queries QN , which retrieve data from XML documents. 2. Optimise all the queries and construct an EUFM for each XML document E based on the queries. 3. Calculate the request at each site for each leaf element. 4. Calculate the pay at each site for each element. 5. Cluster all leaf elements to the site which has the lowest value of the pay to get a set of rooted label paths RP (eV θ ) for each site. 6. Perform vertical fragmentation using the sets of rooted label path and allocate the fragments to the corresponding site. The algorithm first finds the site that has the smallest value of pay and then allocates the element to the site. A vertical fragmentation and allocation schema are obtained simultaneously. An Example. We now illustrate the algorithm using an example. Assume there are five queries that constitute the 20% most frequently queries accessing an XML document from three different sites. 1. πlecturers (Lecturers lecturers//@id= papers//taught Papers) is issued at site 1 with frequency as 20. 2. πtitles, homepage(Lecturers) is issued at site 2 with frequency as 30. 3. πname/lname, phone (Lecturers) is issued at site 3 with frequency as 100. 4. πfname, email (Lecturers) is issued at site 1 with frequency as 50. 5. πtitles, areacode (Lecturers) is issued at site 2 with frequency as 70.

20

S. Hartmann, H. Ma, and K.-D. Schewe Table 1. Algorithm for Vertical Fragmentation QM = {Q1 , . . . , Qm } /* a set of queries elem(e) = {e1 , . . . , en } /*a set of all the leaf nodes of e RP (e) = {rp1 , . . . , rpn } /*a set of rooted label path of all leaf nodes a set of network nodes N = {1, . . . , k} the EUFM of e Output: fragmentation and fragment allocation schema {eV 1 , . . . , eV k } Method: for each θ ∈ {1, . . . , k} let elem(eV θ ) = ∅ endfor for each element ei ∈ elem(e), 1 ≤ i ≤ n do for each node θ ∈ {1, . . . , k} do calculate requestθ (ei ) endfor for each node θ ∈ {1, . . . , k} do calculate payθ (ei ) endfor choose w such that payw (ei ) = minkθ=1 payθ (ei ) elem(eV w ) = elem(eV w ) ∪ ei /* add ei to eV w RP (eV w ) = RP (eV w ) ∪ rpi /* add rpi to RP (eV w ) endfor for each θ ∈ {1, . . . , k}, eV w = πRP (eV w ) (e) endfor Input:

To perform vertical fragmentation using the design procedure introduced in 3 we first construct an Element Usage Frequency Matrix as in Table 2. Secondly, we compute the request for each element at each site, the results of which are shown in the Element Request and Pay Matrix in Table 3. Thirdly, assuming the values of transportation cost factors are: c12 = c21 = 10, c13 = c31 = 25, c23 = c32 = 20, we can now calculate the pay of each attribute at each site using the values of the request in Table 3. The results are shown in a Attribute Request and Pay Matrix in Table 3. Table 2. Element Usage Frequency Matrix

Site

Length 1 2 3

Query Frequency

Q1 Q4 Q2 Q5 Q3

20 50 30 70 100

lecturers lecturer ID name email phone homepage fname lname titles areacode number title 20 20 · 8 20 · 8 2 · 15 · 8 30 · 8 10 20 50 · 8 20 20 20 20 20 20 20 20 0 50 0 0 50 0 0 0 0 0 30 30 0 0 0 30 0 0 0 70 0 70 0 0 0 0 100 0 0 100 100 0

Cost-Based Vertical Fragmentation for XML

21

Table 3. Element Request and Pay Matrix

request/pay ID

name fname lname

request1 (ei ) 20 70 20 pay1 (ei ) 0 0 2800 request2 (ei ) 0 0 30 pay2 (ei ) 200 700 2200 request3 (ei ) 0 0 100 pay3 (ei ) 500 1750 1100 site 1 1 3

lecturers lecturer email phone homepage titles areacode number title 20 70 20 20 20 1000 0 3200 2500 300 100 0 70 0 30 200 700 2200 2200 200 0 0 100 100 0 2500 1750 1900 500 1100 2 1 3 3 2

lecturer1

lecturer2

name

lecturer3

lecturers

lecturers

lecturers

lecturer

lecturer

lecturer

email

name

homepage

name

*phone

ID

fname

STRING

STRING titles

lname areacode

STRING

title

STRING

INTEGER

number

INTEGER

STRING

Fig. 2. XML Tree for the Fragments of Lecturers

Grouping the leaf elements to the site of the lowest pay we get allocations of elements as shown as the last row in Table 3. Correspondingly, we get three sets of rooted label paths that can be used to perform vertical fragmentation: – RP (eV 1 ) = {lecturers/lecturer/(ID, name/fname, email)} – RP (eV 2 ) = {lecturers/lecturer/(name/titles/title, homepage)} – RP (eV 3 ) = {lecturers/lecturer/(name/lname, phone)} When applying projection to the Lecturer column in our table schema, we get three new columns of type XML. The XML schema trees corresponding to these vertical fragments are shown in Figure 2. We now look at how the system performance is changed due to the outlined fragmentation by using the cost model presented above. Assume that the average number of lecturers is 20 and the average number of titles for each lecturer is 2. With the average length of each element given in Table 3, we can compute the total query costs. Two different ways of distributed query evaluation are considered. If distributed XML query processing and optimisation are supported then

22

S. Hartmann, H. Ma, and K.-D. Schewe

selection and projection should be processed first locally to reduce the size of data transported among different sites. In this case, the optimised allocation of document Lecturers is site 2 (see Figure 3) which leads to total query costs of 16,600,000 while the total query costs after the vertical fragmentation and allocation are 4,474,000, which is about one fourth of the costs before the fragmentation. If queries are evaluated in a way that whole documents are shifted to the site issuing the queries to execute the queries over there, then the improvement of the system performance after fragmentation is even more obvious. Before fragmentation, the optimised allocation of XML document Lecturers is site 2, which leads to total query costs of 67,500,000. After vertical fragmentation and fragment allocation, the total query costs are 9,780,000, which is only about one seventh of the costs before fragmentation. This shows that vertical fragmentation can indeed improve system performance. Discussion. In distributed databases, costs of queries are dominated by the costs of data transportation from a remote site to the site that issued the queries. To compare different vertical fragmentation schemata we would like to compare how it affect the transportation costs. Due to the complexity of vertical fragmentation it is practically impossible to achieve an optimised vertical fragmentation schema by exhaustedly comparing different fragmentation schema using the cost model. However, from the cost model above, we observe that the less the value of the pay of allocating an element to a site the less the total costs will be to access it [14]. This explains that the design procedure above can at least achieve semi-optimal vertical fragmentation schema. Using the above algorithm we can always guarantee that the resulting vertical fragmentation schema meet the criteria of correctness rules. Disjointness and completeness are satisfied because all leaf nodes occur and only occur in one of the fragments. Reconstruction is guarantied because all fragments are composed of a set of rooted label path of leaves. Because we use rooted label paths for fragmentation, so that the upper common label path between a pair of vertical fragments serves as a hook for fragments [5]. Using the identification schemata in [5, 25], which use reasonable storage consumption, fragments reconstruction can be in performed in constant time. Before Fragmentation

After Fragmentation

Site 1

Site 1

Q1 Q4

Q1 Q4 f4

f1 f1

f1

f4

E

f1

E2

f2

Q2

E1

f5 Q5

Site 2

f2 Q3 Site 3

E3

f2

f5

Q2

Q5

Site 2

f3 Q3

f5

Site 3

Fig. 3. Allocation before and after Fragmentation

Cost-Based Vertical Fragmentation for XML

4

23

Conclusion

In this paper we presented a vertical fragmentation design approach for XML based on a cost model for measuring total query costs of accessing XML documents. This approach takes into consideration of hierarchical structure of XML documents which are supported by some DBMS as a native data type. Furthermore, a design procedure for vertical fragmentation is presented for XML distribution design. An related problem left for future work is an integrated approach of horizontal and vertical fragmentation.

References [1] Abraham, J., Chaudhari, N., Prakash, E.: XML query algebra operators, and strategies for their implementation. In: IEEE Region 10 Conference, pp. 286–289. IEEE Computer Society Press, Los Alamitos (2004) [2] Andrade, A., Ruberg, G., Bai˜ ao, F.A., Braganholo, V.P., Mattoso, M.: Efficiently processing xml queries over fragmented repositories with partix. In: Ioannidis, Y., Scholl, M.H., Schmidt, J.W., Matthes, F., Hatzopoulos, M., Boehm, K., Kemper, A., Grust, T., Boehm, C. (eds.) EDBT 2006. LNCS, vol. 3896, pp. 150–163. Springer, Heidelberg (2006) [3] Bellatreche, L., Simonet, A., Simonet, M.: Vertical fragmentation in distributed object database systems with complex attributes and methods. In: Thoma, H., Wagner, R.R. (eds.) DEXA 1996. LNCS, vol. 1134, pp. 15–21. Springer, Heidelberg (1996) [4] Bex, G.J., Neven, F., Van den Bussche, J.V.d.: Dtds versus xml schema: a practical study. In: WebDB, pp. 79–84. ACM Press, New York (2004) [5] Bremer, J.-M., Gertz, M.: On distributing xml repositories. In: WebDB, pp. 73–78 (2003) [6] Cornell, D., Yu, P.: A vertical partitioning algorithm for relational databases. In: ICDE, pp. 30–35 (1987) [7] Ezeife, C.I., Barker, K.: Vertical fragmentation for advanced object models in a distributed object based system. In: ICCI, pp. 613–632 (1995) [8] Frasincar, F., Houben, G.-J., Pau, C.: Xal: an algebra for xml query optimization. In: ADC, pp. 49–56 (2002) [9] Goldman, R., McHugh, J., Widom, J.: From semistructured data to xml: Migrating the lore data model and query language. In: WebDB [10] Hoffer, J.A., Severance, D.G.: The use of cluster analysis in physical database design. In: VLDB, pp. 69–86 (1975) [11] Karlapalem, K., Navathe, S.B., Morsi, M.M.A.: Issues in distribution design of object-oriented databases. In: IWDOM, pp. 148–164 (1992) [12] Ma, H.: Distribution design in object oriented databases. Master’s thesis, Massey University (2003) [13] Ma, H., Schewe, K.-D.: A heuristic approach to cost-efficient horizontal fragmen´ Falc˜ tation of XML documents. In: Pastor, O., ao e Cunha, J. (eds.) CAiSE 2005. LNCS, vol. 3520, pp. 131–136. Springer, Heidelberg (2005) [14] Ma, H., Schewe, K.-D., Kirchberg, M.: A heuristic approach to vertical fragmentation incorporating query information. In: Baltic Conference on Databases and Information Systems, pp. 69–76 (2006)

24

S. Hartmann, H. Ma, and K.-D. Schewe

[15] Ma, H., Schewe, K.-D., Wang, Q.: A heuristic approach to cost-efficient fragmentation and allocation of complex value databases. In: ADC, pp. 119–128 (2006) [16] Ma, H., Schewe, K.-D., Wang, Q.: Distribution Design for Higher-Order Data Models. Data. and Knowledge Engineering 60, 400–434 (2007) [17] Miklau, G., Suciu, D.: Containment and equivalence for a fragment of XPath. J. ACM 51 1, 2–45 (2004) [18] Muthuraj, J., Chakravarthy, S., Varadarajan, R., Navathe, S.B.: A formal approach to the vertical partitioning problem in distributed database design. In: International Conference on Parallel and Distributed Information Systems, pp. 26–34 (1993) [19] Navathe, S., Karlapalem, K., Ra, M.: A mixed fragmentation methodology for initial distributed database design. Journal of Computer and Software Engineering 3 (1995) [20] Navathe, S.B., Ceri, S., Wiederhold, G., Dour, J.: Vertical Partitioning Algorithms for Database Design. ACM TODS 9, 680–710 (1984) [21] Navathe, S.B., Ra, M.: Vertical Partitioning for Database Design: A Graphical Algorithm. SIGMOD Record 14, 440–450 (1989) [22] Nicola, M., van der Linden, B.: Native xml support in db2 universal database. In: VLDB, pp. 1164–1174 (2005) ¨ [23] Ozsu, M., Valduriez, P.: Principles of Distributed Database Systems (1999) [24] Schewe, K.-D.: On the unification of query algebras and their extension to rational tree structures. In: ADC, pp. 52–59 (2001) [25] Weigel, F., Schulz, K.U., Meuss, H.: Node identification schemes for efficient xml retrieval. In: Foundations of Semistructured Data (2005)

Efficiently Crawling Strategy for Focused Searching Engine Liu Huilin, Kou Chunhua, and Wang Guangxing College of Information Science and Engineering, Northeastern University Shenyang 110004, China

Abstract. After the analysis and study of focused search engine, we design an efficiently crawling strategy, which include two parts: topic filter and links forecast. The topic filter can filter the web pages having already been fetched and not related to topic; and the link forecaster can predict topic links for the next crawl, which can guide our crawler to fetch topic pages as many as possible in a topic group and when all the topic pages are fetched, it will try to look for another topic group by traversing the unrelated links. This strategy considers not only the precision of fetching web pages but also the recall. With the strategy, the focused search engine can get expected pages without browsing large amount of unnecessary pages, and the user can get the useful information without browsing the huge list which is high recall and low precision. Our extensive simulation studies show that the new strategy has a good efficiency and feasibility.

1 Introduction Information on the Internet is in disorder and be redundant, which limited search efficiency seriously. The appearance of general search engines alleviates the state. However, it is not enough to solve the problems. One side, as the amount of web sites is growing rapidly, the number and size of stored documents is growing even faster and site contents are getting updated more and more often. Although great efforts in hardware and software are done, it is hard to keep up with the growing amount of information available on the web. Study shows, the largest crawls cover only 30–40% of the web, and refreshes take weeks to a month. Another side, the general search engines which are designed for all fields, can fulfill the all-sided search demand. When users input the query, they will return a huge ranked list of the resultant web pages that match the query. The high recall and low precision [1] coupled with this huge list make it difficult for the users to find the information that they are looking for. Such search engine can not meet people’s demand more and more. Another kind of search engine—focused search engine [2] appears. Focused search engine is a new kind of search engine service mode, compared with low queue precision, large amounts information, and not enough depth of general search engine. It provides valuable information and related services for a specific area, a specific group or a specific demand. Its features is “special, refined, deep”, and with industry characteristic which makes focused search engine more focused, specific and in-depth compared K.C. Chang et al. (Eds.): APWeb/WAIM 2007 Ws, LNCS 4537, pp. 25–36, 2007. © Springer-Verlag Berlin Heidelberg 2007

26

H. Liu, C. Kou, and G. Wang

with general search engines massive information disorder. It supplies users with not hundreds of or thousands of related web pages, but extremely narrow and highly specific and targeted information. Therefore it attracts many users. The key part for a good focused search engine is an appropriate crawling strategy [3], by which the crawler will only collect focused information without being bothered by the unnecessary web pages. There are two problems in the crawling strategy: 1. One problem is the precision of calculating the relevance between web pages and topic. Some methods have a high precision, but the strategies are so complex that the systems need many resources to run. Although it is no need of high real-time, it is not worth exchanging the benefit with so much time and space. Otherwise, some strategies have low calculation, but the systems have to sacrifice the precision. How to design an appropriate algorithm is a problem. 2. The other problem is how to predict the potential Url of the links for the next crawl. In the process of fetching the web pages, the existence of web tunnel characteristic [4] impacts on the quality of collection. To improve the accuracy of fetching web pages, we need to increase the threshold, which will lose topic pages group, and the recall decreases. Otherwise, to improve the recall will result in low precision. Therefore, how to balance the recall and precision is really a question. In this paper, we propose an efficiently crawling strategy, which is consist of topic filter and link forecaster. The topic filter is used for filtering the web pages which is not related to the topic, and the link forecaster is used for predicting the potential Urls of the links for the next crawl. The benefit brought by the strategy is much more than the cost, because of no needing to craw the entire Internet. So energy consumption is saved. The contributions of this paper are summarized as follows: 1. We propose an efficient approach which combining the topic filter and the links forecaster. The topic filter can be used to filter the unrelated web pages and save the related web page. The forecaster can be used to forecaster the topic links and find topic links first and consider other links then. 2. Last but not the least, our extensive simulation studies show that the strategy maintains pages on a specific set of topics that represent a relatively narrow segment of the web. It only needs a very small investment in hardware and network resources and yet achieves respectable coverage at a rapid rate. The focused crawling strategy will be far more nimble in detecting changes to pages within its focus than the general search engine’s strategy that is crawling the entire Web. The rest of the paper is arranged as follows. Section 2 briefly reviews the previous related work. And the Section 3 introduces the architecture of focused crawler in which our strategy is applied. Our algorithm is introduced in detail in Section 4. The extensive simulation results to show the effectiveness of the proposed algorithm are reported in Section5. Finally we conclude in Section 6.

2 Related Work The concept of focused search engine has appeared since the general search engine turned up. Therefore, there was focused crawl strategy called Fish-Search [5]

Efficiently Crawling Strategy for Focused Searching Engine

27

designed by De Bra et al. at that time. In Fish-Search, the Web is crawled by a team of crawlers, which are viewed as a school of fish. If the ‘‘fish’’ finds a relevant page based on keywords specified in the query, it continues looking by following more links from that page. If the page is not relevant, its child links receive a low preferential value. But it is difficult to assign a more precise potential score to the pages which haven’t been yet fetched. Shark-Search [6] by Hersovici et al. did some improvements to the original Fish-Search algorithm. Shark-Search is a modification of Fish-search which differs in two ways: a child inherits a discounted value of the score of its parent, and this score is combined with a value based on the anchor text that occurs around the link in the Web page. Besides the kind of methods above, there is another kind of methods which use a baseline best-first [7] focused crawling strategy combined with different heuristics. For example, C. Aggarwal, F. Al-Garawi and P. Yu designed an intelligent crawling with arbitrary predicates [8]. They propose the novel concept of intelligent crawling which actually learns characteristics of the linkage structure of the World Wide Web while performing the crawling. Specifically, the intelligent crawler uses the inlinking web page content, candidate URL structure, or other behaviors of the inlinking web pages or siblings in order to estimate the probability that a candidate is useful for a given crawl. Also, the system has the ability of self-learning, i.e. to collect statistical information during the crawl and adjust the weight of these features to capture the dominant individual factor at that moment. There are other methods used to capture path information which is help lead to targets such as genetic programming. For example, a genetic programming algorithm is used to explore the space of potential strategies and evolve good strategies based on the text and link structure of the referring pages. The strategies produce a rank function which is a weighted sum of several scores such as hub, authority and SVM scores of parent pages going back k generations [9]. Currently, the one thing people focus on is how to judge whether a web page is a topic page. The method designed can judge the web pages at certain accuracy, and massive time and space is not needed. And the other thing people focus on is how to recognize the potential Urls for the next crawl. Many methods designed before, think the links of a related page tend to topic. It is true, but not all the links is topic-related. If fetched all the pages corresponding to the links, a lots of unrelated pages will be fetched back, which will affect the precision. The links of an unrelated page shouldn’t be discarded entirely, because there are some links which may point to a topic page. Even if the links don’t point to a topic page, traversing some unrelated links can reach a topic group, which plays an important role in ensuring the recall. Therefore, the current research is the appropriate topic filter and a good link forecaster which forecast the potential Urls, and ensure at certain precision and recall.

3 Architecture of Crawling for Focused Search Engine The web crawling for focused search engine is done by a focused crawler, which adds a topic relevance determinator, based on a general crawler [10]. Its structure can be described by figure 1.

28

H. Liu, C. Kou, and G. Wang

Internet

Crawl Component

Urls Queue for Crawl

Parse Component Topic relevance determinator Topic Filter

Links Forecaster

Database of Pages

Database of Urls

Fig. 1. Architecture of Crawling

From figure1, we can know the focused crawler is constituted of several important parts such as crawl component, parse component and topic relevance determinator. The crawl component fetches the web pages according to the Urls queue coming from the database of Urls. The parse component parses these pages. The topic relevance determinator analyze the pages which have been parsed, and determine which pages should be preserved and which links should be added into the Urls queue for the next crawl. The database of pages is used for saving the topic pages, and the database of Urls is used for saving all the Urls of links. With the architecture it works like this: First, inject a set of Urls selected from the authority sites of certain special fields by manual into the database of Urls, and links forecaster will add these Urls to the Urls queue. Then the crawl component fetches the corresponding pages. After that, these pages will be parsed by the parse component. The html page information like title, anchor, and text and so on, is available from the parsing results. Making full use of the information, system will analyze the pages’ relevance, and save in database of pages or throw some of them by the different relevance through the topic filter, one part of topic relevance determinator. Then update the database of Urls and forecast the potential Urls for the queue for crawl through the other part of topic relevance determinator. A new cycle begins.

4 Crawling Strategy of Focused Search Engine From the architecture of the focused crawler, the topic relevance strategy [11] contains the topic filter and the links forecaster. The topic filter calculates topic relevance by classifying based on vector space model, and then decides whether web pages will be preserved or not, according to the value of the topic relevance. The links forecaster predicts topic relevance of linked web pages, when no previous knowledge of link structure is available, except that found in web pages already fetched during a crawling phase. So the links forecaster needs to get the useful information from the web pages already fetched. For example, the forecaster can get some apocalypse from the

Efficiently Crawling Strategy for Focused Searching Engine

29

topic relevance of the page in which links are and from the existing link structure in the database of Urls. Then the focused crawler can be guided by such a strategy that predicts first, and judges later. And the strategy can fetch topic pages at certain precision and recall, and not need to crawl the entire web. 4.1 The Topic Filter The task of the topic filter is to judge the similarity between the topic and the page having already been fetched. In fact, the calculation of the content topic relevance also can be considered as a process of classification to pages. The first thing that must be done is to convert the web page into an expression form that makes the web pages can be recognized by computers. There are some models based on keywords such as Boolean model, vector space model and probability model. So far, the vector space model is applied widely and that’s the reason why we adopt it. The vector space model (VSM) [12]uses the characteristic item set which appears in the web pages to replace them, so the original pages can be expressed by the vector. The strategy designed by us just expresses the web page texts using this kind of model. Then, a text D can be represented as: D={(T1,W1), (T2,W2), (T3,W3)……(Tn,Wn)} In the formula, Ti is the characteristic item, Wi is the weight of ith characteristic item (n=1, 2 ….). Words appearing on one page are considered as the characteristic item. Therefore, the first important work is extracting the characteristic item from web pages. The focused search engine aims at crawling web pages in certain specific domain, and characteristic items in this kind of web pages generally are the technical terms. So we do not need to do segmentation and frequency statistics [13] to the ordinary words. In order to improve the efficiency of whole system and the accuracy of segmentation, we use the topic dictionary to help finish segmentation of web pages. The topic dictionary is determined by experts in the field. The different logic division of the text will make the characteristic items play a different role. When one characteristic item vector is used to describe the content of the text logic passage for every text logic passage, matching search considers not only the number of the characteristic items but also the location. The method can improve the precision of matching results greatly. Therefore, since the web pages have been expressed in different logic part, we can’t only adopt the frequency of characteristic item as the weight. In one web page, text in Head label, the anchor text and the ordinary text in Body label nearly show the whole information, so they can be used while calculating the weight of characteristic items. We design the following algorithm to get the weight. Hweight is the weight of Head label in web page; Aweight is the weight of Anchor label in web page; Bweight is the weight of Body label in web page; and satisfy Hweght+Aweight+Bweight=1; If the characteristic item appears in a certain label, it will be recorded as Focused (label) =true, otherwise recorded as Focused (label) =false. Therefore, the weight of a characteristic item is calculated as follows:

30

H. Liu, C. Kou, and G. Wang

Initialize weight=0; If(Focused(Head)==true) Then weight+=Hweight; If(Focused(Anchor)==true) Then weight+=Aweight; If(Focused(Body)==true) Then weight+=Bweight; After transforming the web page text into the vector space model using the topic dictionary and the segmentation technology, we’ll select a typical topic web page as the benchmark-vector. Then it is feasible to calculate the similarity [14] of web pages. For example, similarity between the web page text i and the base-vector j can be expressed as: n

D S im ( D i , D j ) =

∑

(W

k =1

ik

n

∑

k =1

,W

jk

)

n

W

2 ik

*

∑

k =1

W

2

(1)

jk

According to formula (1) we can calculate the similarity between one web page and benchmark-vector. We can get a threshold value through experiments. The page whose value is higher than this threshold belongs to this topic, and will be preserved. Otherwise, the page will be discarded. The value of similarity will also provide important information for the links forecaster. 4.2 The Links Forecaster The topic filter can filter the web pages which are not related to the topic. However, if we capture all pages down from the web and then judge whether we will save the page, we’ll waste massive time and space. Therefore, it is necessary to forecast topic relevance according to the web pages which have been fetched. Our goal is to crawl related web pages as much as possible and to crawl unrelated web pages as little as possible. Therefore, we need analyze the features of web pages which will be helpful for predicting the topic links, and then design a better strategy. 4.2.1 The Analysis of Web Features First, the labels of a web page include the topic features. An important label is the text around links such as anchor text. They are the descriptions to pointed web pages. If the topic word turns up in these texts, the page that the link points to is likely the topic page. For example, if a web page includes such a label: agriculture, the pointed page is related to agriculture possibly. Second, web is a hypertext system based on Internet. The biggest difference between hypertext system and ordinary text system is that the former has a lot of hyperlinks. Study shows, hyperlinked structure on the web has contained not only the characteristic of page importance, but also the topic relevance information. Because of the distribution characteristic of pages on the Web, we can have such conclusions: (1) Page A pointed by Page B has a good relevance with the topic of A. (2) Page linked by a topic page will be a worthy one. Therefore, we calculate the score of a page by PageRank [15, 16]. The Algorithm has an effective use of huge link relations in the Web [17].It can be described as: one link from A pointing to B is considered as an A’ providing vote to B, then we can judge the page importance according to the number of vote. The formula [18] is:

Efficiently Crawling Strategy for Focused Searching Engine

PR (A) = (1-d) + d (PR (T1)/C (T1) + ... + PR (Tn)/C (Tn))

31

(2)

In formula (2), PR(A) The PageRank value of page A PR(Ti) ThePageRank value of Ti Pointing to page A C(Ti) The number of out links of page Ti d Damping d ∈ (0,1)

：

： ： ：

；

（

）

；

；

4.2.2 The Prediction Algorithm Definition: There are three attributes in every Url in the database of Urls. One is score which is used for storing the value calculated by the topic filter. The value of score represents the extent similar to the topic. The bigger the value is, the web page tends to the topic more. Otherwise, the smaller the value is, the web page is further from the topic. Another is pscore which is used for storing the value calculated by PageRank according to the score. The value of pscore represents the linked page’s extent similar to the topic. The bigger the value is, the linked page tends to the topic more. Otherwise, the smaller the value is, the linked page is further from the topic. The last one is fetch flag which is used for the information whether the page has been fetched. If the value of the fetch flag is true, it represents that the page corresponding to the Url has already been fetched, otherwise, it hasn’t been fetched. Initiation: Inject the topic Urls into the database of Urls and initiate all the score=0, all the pscore=0, all the fetch flag=false. Initiate fetching turn=n, topic threshold=t, number of urls in Urls queue for crawl=N. Then add all the Urls in the database of Urls to the Urls queue for crawl. Description: First, judge whether the fetching turn is reached, if not reached, fetch the web pages corresponding to the Urls of the Urls queue. The topic filter will calculate the similarity value of every page. Then the forecaster will do two things: one is to update the database of Urls, the other thing is to forecast the potential links for the next crawl. During updating the database, there are four things to do: 1. The fetch flag of the Url whose pages having already been fetched should be set true. The purpose to do so is that the system won’t fetch these pages later. 2. The Urls extracted from the fetched page by the parser should be added into the database of Urls and the link relation between each other should also be stored. The purpose to do so is that it can form a hyperlinked structure that will be used while forecasting the potential links. 3. If the anchor text of the link includes topic words, the pscore of the Url of the link should be set 1.0. Because once the topic words turn up in anchor text, the link may point to a topic page, the pscore of the Url should be set as high as 1.0. The Url will likely exist in the Url queue in the next cycle. 4. If the similarity value larger than the topic threshold, the score of the Url should be set the value of the similarity. Otherwise, the pscore of the Url of every link in the page should be set 0.5*p. Here p is a random probability. The links in unrelated page shouldn’t be discarded, because we may traverse some such links and find many topic web pages which get together. The benefit is so large that we must consider. During the forecasting, there are four things to do:

32

H. Liu, C. Kou, and G. Wang

1. According to the links relation that has existed in the database of Urls, compute the PageRank value. Every page whose topic relevance value is higher than the threshold does a division to the topic relevance value. Give every share to the links of the page. Iterating in such way, every Url will have a PageRank value. 2. Normalize the PageRank by ePageRank_value-e-PageRank_value/ ePageRank_value+e-PageRank_value. If the normalization larger than the original pscore of the Url, replace it. 3. Sort the Urls according to the value of pscore, top N Urls are considered to be the Urls that tends to the topic and added to the queue for crawl. The steps are shown in the Algorithm 1. Algorithm 1. Forecast links 1: While (fetching turn is not reached) 2: Fetch the web pages corresponding to the Urls. 3: The similarity value of every page is calculated by the topic filter. 4: Update the database of Url, do 5, 6, 7, 8, 9; 5: Set the fetch flag of the Url whose pages having been fetched=true. 6: Add the Urls of the links in the page to the database and store the link relation. 7: If the anchor text of the link include topic words, set the pscore of the Url of the link=1.0. 8: If the similarity value larger than topic threshold, set the score of the Url =the similarity. 9: Else set pscore of the Url of the link=0.5*p (p is random probability) 10: Forecast the potential links, do 11, 12, 13,14; 11: According to the link relation, compute the PageRank (PR (T) is the score) value of every Url. 12: Normalize the PageRank value, compute the normalization by ePageRank_value-e-PageRank_value/ ePageRank_value+e-PageRank_value 13: If the pscore larger than the original pscore of the Url, replace it. 14: Sort the Urls according to the value of pscore, top N Urls are added to the queue for crawl. 15: EndWhile

5 Experiments The algorithm is implemented with Eclipse and running Linux Professional Edition. The experiments are conducted on a PC with InterPentium4 2.8GHz CPU, 1G main memory and 80G hard disk. We conducted experiments with the real pages on Internet. To examine the effects of topic-first strategy, we use two evaluation benchmarks [19]: precision, and recall. Precision is the ratio of information from given algorithm and result by manual categorizing. Recall is the ratio of correct information that classifies by an algorithm and the precise information with reality. For this algorithm: Precision =expPage/toreturnPage;

(3)

Recall=expPage/existPage;

(4)

Efficiently Crawling Strategy for Focused Searching Engine

33

In the formula (3) and (4), expPage: the number of correct expected page, toreturnPage: the total number of returned pages actually, existPage: the number of expected pages. In this experiment, 1000 agricultural web pages are disposed for text training, in order to extract and choose characteristic as the base-vector of this topic. The number of topic web pages, which exist on the Internet, is estimated by the method of statistical sampling. Then we crawl the Internet respectively with common search engine and focused search engine for agriculture. The common search engine adopts the breadth-first as the crawling strategy, while the focused search engine for agriculture adopts the topic-first as the crawling strategy. Experiment 1: We start with 100 seed Urls which are all topic Urls, set the threshold 0.3, and crawl with the increase of link-depth .Then we can draw the figures below:

Fig. 2. Precision of topic-first and Breadthfirst with topic seed Urls

Fig. 3. Recall of topic-first and breadthfirst with topic seed Urls

The Carves in Fig. 2 show the precision of two different strategies. From this figure we can know, the topic-first strategy has a high precision in the first round, due to the Urls we have injected are all about agriculture. In the process of crawling, the topic strategy will filter unrelated web pages, while the breadth-first strategy will crawl all pages, so it has a low precision compared with topic-first. With the increase of link-depth, precision of two methods decline and tend to be stable. Comparing the two carves, we can easily find out that the agriculture focused crawling strategy is superior to breadth-first obviously in precision. And Fig.3 compares the recall value of two strategies. From Fig.3, we can know after crawling 5000 pages, the recall value of agricultural focused crawling strategy ups to 44.1%, and the breadth-first only is about 11%. Experiment 2: We start with 100 seed Urls which inlcluding 50 topic Urls and 50 unrelated Urls, set the threshold 0.3, and crawl with the increase of link-depth .Then we can draw the figures below:

34

H. Liu, C. Kou, and G. Wang

Fig. 4. Precision of topic-first and breadthfirst with not all topic seed Urls

Fig. 5. Recall of topic-first and breadthfirst with not all topic seed Urls

The Carves in Fig. 4 show the precision of two different strategies with seed Urls not all being topic Urls. They all decline until stable state. Compared with Fig.3, the precision is apparently lower, because the initiation seed Urls is not all topic Urls, the forecaster will add unrelated Urls to the Urls Queue when there are no topic Urls. In this way, the effect of our topic-first strategy will be a little worse. Therefore, to make the strategy work well, better seed Url should be selected. Fig.5 shows the recall of topic-first and breadth-first with seed Urls not all being topic Urls, we can know after crawling 5000 pages, the recall value of agricultural focused crawling strategy ups to 34.6%, and the breadth-first only is about 9.7%. Although lower than the topic-first in Fig.5, the precision of the topic-first is still better than the breadth-first. Experiment 3: We start with 100 seed Urls which are all topic Urls, set the threshold 0.2, and crawl with the increase of link-depth .Then we can draw the figures below:

Fig. 6. Precision of topic-first and breadthfirst with a different threshold

Fig. 7. Recall of topic-first and breadthfirst with a different threshold

The Carves in Fig. 6 show the precision of two different strategies with a different threshold. They all decline until stable state. Compared with Fig.3, the precision is a little lower, because the threshold is set lower, a lot of unrelated web pages may

Efficiently Crawling Strategy for Focused Searching Engine

35

be judged as the topic pages, which will misdirect the links forecaster to add the unrelated Urls to the Urls queue. Fig.7 shows the recall of topic-first and breadth-first with a different threshold, we can know after crawling 5000 pages, the recall value of our topic-first strategy ups to 48.9%, and the breadth-first is only about 11%. The reason why the recall value of our topic-first strategy is higher than that in Fig.3 is the lower threshold. Although lower threshold results in lower precision, it can improve the recall. In these experiments, after the same time, the number of the topic pages having already been fetched with our topic-first strategy is more than that of the breadth-first with the fewer web pages fetched. It means a lot of time and space is saved. Therefore, it is obvious that the topic-first strategy works better than breadth-first in both precision and recall. All these results reflect the great advantage of topic-first crawling strategy. And to keep the effect of our topic-first strategy, good seed Urls and appropriate threshold should be selected.

6 Conclusion Crawling strategy of focused search engine combines the topic filter and the links forecaster. In the part of topic filter, we uses the method of classification based on the vector space model, which has many good point such as clear expression, convenient, smaller computation complexity. While extracting the characteristic items, a certain domain topic dictionary can cover some topic well use few words, and make the words quantity and complexity reduce greatly. While computing the weight of the characteristic items, we consider the feature of the web page and improve the precision. Therefore, with the process of topic filter, the system can filter out the web pages which have nothing to do with the topic. In the part of the links forecaster, we have used the anchor text and the link structure analysis method, which allows us to get the topic web pages without crawling entire Internet. This strategy has been used in the focused search engine for agriculture designed by us; at present it works well, and has an appropriate precision and recall ratio for search results. Our following work is: determine the best parameter and the threshold by the massive experiments, do further research and continually optimize the strategy.

Acknowledgement This work is supported by the national natural science foundation under the grant No. 60573089.

References 1. Crestani, F., Lalmas, M., van Rijsbergen, C.J.: Information Retrieval: Uncertainty and Logics[M]. Kluwer Academic Publishers, Boston, Massachusetts (1998) 2. Chakrabarti, S., van den Berg, M., Dom, B.: Focused crawling a new approach to topicspecific Web resource discovery[J]. Computer Networks 31(11-16), 1623–1640 (1999)

36

H. Liu, C. Kou, and G. Wang

3. Pant, G., Tsioutsiouliklis, K., Johnson, J., Giles, C.: Panorama: extending digital libraries with topical crawlers[C]. In: Proceedings of ACM/IEEE Joint Conference on Digital Libraries, Tucson, Arizona, June 2004, pp. 142–150 ( 2004) 4. Bergmark, D., Lagoze, C., Sbityakov, A.: Focused crawls, tunneling, and digital libraries[C]. In: Proceedings of the 6th European Conference on Digital Libraries, Rome, Italy (2002) 5. Bra, P.D., Post, R.: Information retrieval in the World Wide Web: making client-base searching feasible[C]. In: Proceedings of the 1st International WWW Conference, Geneva, Switzerland (1994) 6. Hersovici, M., Jacovi, M., Maarek, Y., Pelleg, D., Shtalhaim, M., Ur, S.: The Shark-search algorithm-an application: tailored Web site mapping[C]. In: Proceedings of the 7th International WWW Conference, Brisbane, Australia (1998) 7. Menczer, F., Pant, G., Srinivasan, P.: Evaluating Topic-Driven Web Crawlers[C]. In: Proceedings of the 24th annual international ACM SIGIR conference on Research and development in information retrieval, (September 2001) 8. Aggarwal, C., Al-Garawi, F., Yu, P.: Intelligent crawling on the World Wide Web with arbitrary predicates[C]. In: Proceedings of the 10th International WWW Conference, Hong Kong (2001) 9. Johnson, J., Tsioutsiouliklis, K., Giles, C.L.: Evolving strategies for focused Web crawling [C]. In: Proceedings of the 20th International Conference on Machine Learning (ICML2003), Washington, DC, USA (2003) 10. Liang, Xiaoming, L., Xing, L.: Discussion on theme search engine[A]. In: Advances of Search Engine and Web Mining[C], pp. 34–40. Higher Education Press, Beijing (2003) 11. Can, F., Nuray, R., Sevedil, A.B.S.: Automatic performance evaluation of Web search engines. Information Processing & Management[J] 40(3), 495–514 (2004) 12. Ming, L., et al.: Improved Relevance Ranking in Web-Gather[J]. Comput. Sci. & Technol. 16(5), 410–417 (2001) 13. Hong, C., Jinsheng, Y.: Search on Forestry Focused Search Engine [J]. Computer Applications, 24 ( 2004) 14. Gianluigi, G., Sergio, G., Ester, Z.: A Probabilistic Approach for Distillation and Ranking of Web Pages[C]. World Wide Web, pp. 1386–145X ( 2001) 15. HuanSun, Wei, Y.: A note on the PageRank algorithm. Applied Mathematics and Computation[J] 179(2), 799–806 (August 2006) 16. Higham, D.J.: Google PageRank as mean playing time for pinball on the reverse web. Applied Mathematics Letters[J] 18(12), 1359–1362 (December 2005) 17. Almpanidis, G., Kotropoulos, C., Pitas, I.: Combining text and link analysis for focused crawling-An application for vertical search engines[M]. Information Systems Article in Press, Uncorrected Proof, Available online November 7, 2006 (2006) 18. Brin, S., Page, L.: The anatomy of a large-scale hypertextual web search engine[C]. In: Proceedings of the Seventh International World Wide Web Conference,Brisbane, Australia (1998) 19. Chekuri, C., Goldwasser, M., Raghavan, P., Upfal, E.: Web search using automatic classification [C]. In: Proceedings of the Sixth International WWW Conference (1997)

QoS-Guaranteed Ring Replication Management with Strong Consistency* Wei Fu, Nong Xiao, and Xicheng Lu School of Computer, National University of Defense Technology, Changsha, P.R. China [email protected]

Abstract. As an important strategy for reducing access time, balancing overload, and obtaining high data availability, replication is widely used in distributed system such as Data Grids. Currently grid researchers often consider replica placement problem from the view of whole systems, i.e, ignoring individual requirements. With the rapid growth of Data Grid applications, fulfilling requirements of every request becomes a big challenge, especially in Data Grids with massive data, multiple users and geographically distribution. We build up a ring-based Data Grid model and define a QoS-guaranteed Replica Management Problem. A QoS-guaranteed replica placement algorithm qGREP is proposed, which takes individual QoS requirement into account and achieves tradeoff between consistency cost and access efficiency. Also a ring-base electing algorithm VOSE is presented to solve update conflicting while minimizing propagating costs. Analysis and experiments show that the model and algorithm are feasible in complex Data Grid environment and achieve good performance under different simulating conditions.

1 Introduction Data Grid[2, 3] employs Grid technologies to aggregate massive storage capacity, network bandwidth and computation power across widely distributed network environment to provide pervasive access to very large quantities of valuable data resources. As an important strategy for reducing access time, balancing overload, and obtaining high data availability, replication is widely used in distributed system such as distributed file system[4-6], Web caching[7-9], database management and Data Grids[2, 10-13]. Replica placement problem[14] is an important function of replica management, which defined as: given M sites and a data objects, find a subset of sites R to place a replica of the data onto each of them, obtaining some benefits. For example, traditional researches aimed at minimizing the total/average costs as much as possible. The cost can be accessing delay, communication message, or response time. However, this criterion cannot be valid in some circumstance. In bank or telecom systems, the total/average response time is required to be as low as possible. At the same time, the response time of EACH request is also required below some service quality requirements. These two *

This paper is supported by National Basic Research Program (973) NO. 2003CB317008, NO. 2005CB321801, and Chinese NSF NO. 60573135, NO. 60503042.

K.C. Chang et al. (Eds.): APWeb/WAIM 2007 Ws, LNCS 4537, pp. 37–49, 2007. © Springer-Verlag Berlin Heidelberg 2007

38

W. Fu, N. Xiao, and X. Lu

ways have different objectives to place replica, although in many cases they have almost the same solutions. In this paper we will consider the replica placement from the viewpoint of individuals, each holding a pre-defined QoS requirement. Data objects are often supposed read-only to cater to the requirement of today’s grid applications. Unfortunately writing in grid has not been well solved yet and still remains as an unavoidable challenge. In a writable system, consistency is very important for the correctness and efficiency of data access. We would like concentrate our discussion on maintaining the consistency of data content while guaranteeing the access efficiency. Generally speaking, access efficiency and consistency cost are two conflicting goals. More replicas obtain better access efficiency by bringing data to the vicinity of more users, but the cost of update will increase correspondingly. And vice versa. Therefore a suitable trade-off should be made for both satisfactions. In a writable Data Grid environment, we consider both query & update request and present a ring based model to manage replica consistently. With a pre-description of QoS, a ring structure with sufficient replicas will be produced to meet individual access requirements, meanwhile minimizing the cost of update. An electing algorithm is proposed to harmonize simultaneous update to achieve strong consistency. The paper is organized as follows. In Section 2 we propose a ring-based model to depict a QoS-guaranteed replica management. In Section 3, an algorithm is presented to solve the replica placement problem under the proposed model. In Section 4, a varietal electing algorithm is provided for strong consistency. Section 5 includes experiments which show some characteristic of our model. After a introduction of related work in Section 6, we give a brief conclusion and some future work at Section 7.

2 Ring-Based Data Grid Model Typically, Data Grid is described as a combination of Computational Elements (CE) and Storage Elements (SE) [10]. Every data access will be directed to some SE eventually, either from CE or SE. We regard the access from CE to SE as coming directly from the SE being accessed. Thus all CEs can be omitted and all SEs make up an overlay storage graph. We propose a ring-based Data Grid model as follows: Definition 1. A Ring-based Data Grid RDG={G, DM, D, ξ,s, R}, where - G: is a network of storage nodes, represented by G=(V, E), where V is the set of all storage element nodes, and E∈V×V is the set of direct links between two nodes. Each edge has communication delay as weight; - DM: is a matrix holding shortest communication costs of all node pairs in G; - D: represents the diameter of G, measured by network delay; - ξ: stands for a proportion of update ξ(0≤ξ<1), this value is a statistics from accessing history in a given time window. Here we assume that accesses are partitioned into query from ordinary node to replica node and update from one replica node to all the other replica nodes; - s: is the node id where the object residing on, here we assume each node has a global unique distinguished id; - R: represents for a set of nodes, each of them replicate data from s to its local disk. Nodes in R are organized as a unidirectional ring call replica ring;

QoS-Guaranteed Ring Replication Management with Strong Consistency

Definition 2.

∈

39

∈

∀ u, v V, the cost between u and v is defined as:

- If (u,v) E, then d(u, v) equals to the delay of the link between u and v. - Otherwise, d(u, v) equals to the total delay of the shortest path between u and v. Note: A given cost can be considered as relatively stable. If not, it can be replaced by an average value of historical observations. A storage graph is illustrated as follows by Fig. 1.

(a)

(b)

Fig. 1. (a) is a storage graph of 15 nodes with random distribution topology, D = 48 and ξ= 0.2. Source data resides on site labeled by 0. (b) is a replica ring generated from (a). Dashed lines in (b) represent the propagating paths when updates take place. It guarantees that query cost from any node is no more than 16, while update cost is also minimal.

In Fig. 1.(a) each grey node represents for a replica node with a replica residing on it. Let R be the set of replicas. White node represents for ordinary node, which can turn into a replica node by fetching data to its local disks. Fig. 1.(b) shows a unidirectional replica ring composed by three replica nodes. There are logical communication path between two neighbor node pair. The number of nodes in the ring is constricted by the QoS requirement, and this will be discussed below. In reality, the total cost of a typical cost includes the delay of network, overhead of data transferring and other factors. In order to simplify the problem, we only consider the communication delay. In our model, any two nodes can communicate with each other directly or indirectly, i.e., there exists no isolated node. Otherwise, it can be removed from the graph easily. The storage capacity of each node is assumed to be infinite, due to the lower and lower price of disks. To say the least, a replica replacement policy can partly solve the problem of volume exceeding. Besides the infinite capability of storage, each node has little but enough computational capability to run some simple protocols. Through some sort of grid monitoring and/or information service[1], global information can be sent to every node for the preparation of our algorithms VOSE and qGREP, as to be discussed below.

3 QoS-Guaranteed Replica Placement and qGREP Algorithm Replica promotes access efficiency and decrease total query cost by placing data to users’ vicinity. With the storage graph and the ring-based replica model, in this

40

W. Fu, N. Xiao, and X. Lu

section we discuss about replica placement with QoS guarantee. We want to find a tradeoff between query and update cost, while guaranteeing that every node could obtain data within a QoS restriction and overall accessing cost can be decreased as much as possible This deducts the following QoS-guaranteed Replica Management Problem: Definition 3. Given a ring-based Data Grid RDG with G, DM, ξ, and s, a QoSguaranteed Replica Management Problem (QRMP) is to find a set of nodes R to replicate data from s to R, promising that each query can be fulfilled within a pre-defined QoS restriction Q, while update cost is minimized. We present a central algorithm to find a minimal replica set for a given storage graph and QoS restriction, as depicted by the pseudo-code in Fig. 2. Input: G =(V, E); DM; ȟ; Q; s Output: R: init {s}; 1 Begin 2 D MAX{DM [i, j ]} ;// the diameter of G 1di , j d n

3 4

5 6 7 8 9 10 11 12 13

If Q<¯*D/(1-¯) // qualification inequation, see Sec. 3.2 Warning! //run a risk, set Sec. 3.2 Dquery=Q + 1; //current max query cost, initially greater than Q CV=DM[s]; //vector of the minimum cost to replica ring While Dquery > Q, loop: // see Sec. 3.1 For each row vector DM[i]=DM[i,.] in DM, loop Profit Vector:PV[i] = GetProfit(CV, DM[i]); Find a row making the biggest PV[i], let it be i'; Add2Ring(R, i'); CV = Merging(CV, DM[i']); Dquery˙ MAX {CV[i]}; 1d i d n

14 End of loop 15 End //normal terminating

Fig. 2. Centralized algorithm: QoS-Guaranteed Replica Placement (qGREP)

Line 9: GetProfit(V1,V2) returns the sum of positive differences of V1 and V2. If V1[i]> V2[i], the minus of (V1 [i]- V2[i]) is added into PV[i]; otherwise omits it. Line 11: Add2Ring(R, i') adds the selected node i' into the replica ring R. First it computes the sum of costs to any two neighbored node pair in the ring. Then it chooses the pair with the smallest sum and inserts itself between them. Line 12: Merging(V1,V2) picks out the smaller element of V1[i] and V2[i] as the new V1[i]. It can be easily deduced that qGREP has a computational complexity of Θ(n3) at the worst case, and a space complexity of Θ(n2). 3.1 Origination and Principle qGREP considers both query and update cost at the same time. Distinctly, more replicas mean less total/individual query cost. But more replicas mean more update cost,

QoS-Guaranteed Ring Replication Management with Strong Consistency

41

including higher probability of update conflict and more propagation overhead. Therefore the basic strategy or principle is to give a optimal tradeoff between them. As illustrated in Figure 1, each node in V will find a nearest replica in R for better access latency. After a new replica u is inserted between a pair (i, i+1) in the ring, query cost decrement is calculated as follows: Δ

Query(u)=

∑

MIN ( d (v, r )) -

v∈(V − R )

r∈R

∑

MIN (d (v, r ))

v∈(V − R )

(1)

r∈RU{u}

Meanwhile, the update cost will increase at two aspects. Firstly, the chance of update conflict will increase. Secondly, the propagation of update will suffer longer latency among more replica nodes. Let replicas number be n, the probability of update conflict should be: Pn= 1 − C n (1 − ξ ) 0

n

− Cn1ξ (1 − ξ ) n−1 = 1 − (1 − ξ ) n − nξ (1 − ξ ) n−1

(2)

After u added, the probability should be: Pn+1=1-

Cn0+1 (1 − ξ ) n +1 - Cn1+1ξ (1 − ξ ) n =1- (1 − ξ ) n +1 - (n + 1)ξ (1 − ξ ) n (3)

Hence we calculate the difference of update cost as: Update(u)

P

MAX

P

MAX

(4)

Where v is the update initiator. Thus the total changing of access cost should be: Cost

Update

(5)

Positive ΔCost means that adding replica induces more query cost decrement than update cost increment. Therefore the total access cost decreases. And vice verse. When adding replica does harm to the overall performance, no more replica should be added. In Sec. 3.2 we will prove that qGREP always make positiveΔCost under some conditions. Therefore, qGREP should stop as soon as every query can be served within a pre-defined QoS restrict. 3.2 Correctness and Qualification In this section we first give proof that with correct QoS constrict for a given storage graph, qGREP will produce a solution with low time complexity. Recall the pseudo-code of qGREP. Essentially, AV records the smallest costs from every node to access replica ring, and Dquery is the biggest one. When qGREP terminated normally, any element of AV must be smaller than Q. Thus the QoS restrict is fulfilled. On the other hand, once qGREP notices that QoS restrict is fulfilled, it will terminate immediately, and the dead lock of no-stop-looping would never occur. Furthermore, PV[i] stands for the possible decrement of query cost if node i becomes a replica node. Thus qGREP always promotes the most “valuable” node to hold new replica.

42

W. Fu, N. Xiao, and X. Lu

Next we will prove that under suitable conditions, qGREP will always decrease total access cost when adding new replica. Lemma 1. Add a new replica node u, the difference of total query cost should be: Query(u) ≥ MIN (d (u, r )) >0.

Δ

r∈R

Nodes in node u’s vicinity may turn to access this new replica if their query costs are less than before, thus the total cost will be decreased. The worst case is that no such node exists, however this will still eliminate the query latency from u itself, i.e. MIN ( d (u , r )) . r∈R

Lemma 2.

∀ 0<ξ<1, the conflict probability will increase: Pn+1 >Pn >0. Furthermore,

ΔUpdate(u)>0.

Proof. From the meaning of Pn we know Pn >0. For any integer n>1, 0<ξ<1, Pn+1 -Pn = (1-ξ)n + n ξ (1-ξ)n-1- (1-ξ)n+1- (n+1) ξ (1-ξ)n =n ξ2 (1-ξ)n-1>0, so it can be deduced easily that Pn+1 >Pn >0. From (4) and (6),

Δ Update(u)>

(6)

Pn ( MAX ( d ( r , v)) - MAX (d (r , v )) ). Since for any r∈R ∪{u }

v∈R

■ Theorem of Qualification. If QoS restrict Q meets the inequation: Q>ξD/(1-ξ), update initiator v, MAX ( d ( r , v )) > MAX (d (r , v )) , thus ΔUpdate(u)>0. r ∈R ∪{u }

v∈R

where D is the network diameter of storage graph G, adding a node to replica ring will always decrease the total access cost, i.e., ΔCost > 0. Proof. Substitute ΔQurey and ΔUpdate in (5) with (1) and (4) :

Δ

ξ

Cost = (1- ) (

∑

v∈(V − R )

MIN (d (v, r )) r∈R

∑

v∈(V − R )

MIN (d (v, r )) )-

r∈R U{u }

ξ

(Pn+1

MAX ( d ( r , v )) - Pn MAX (d (r , v )) )

ξ P ( MAX (d (r, v)) - MAX (d (r, v)) ) Since MIN ( d (u , r )) ≥ Q, and MAX ( d ( r , v )) - MAX (d (r , v )) ≤D, Q>ξD/ (1-ξ), Δ Cost >(1-ξ) Q-ξD> ξD-ξ D =0 ■ r ∈R ∪{u }

ξ

v∈R

> (1- ) MIN (d (u , r )) r∈R

r∈R

r ∈R ∪{u }

n

r ∈R ∪{u }

v∈R

v∈R

Theorem of Qualification reveals the internal relationship between network topology and access model. It stands for the trade-off point of QRMP problem. Therefore, we can use the inequation to judge the existence of feasible solution. When it is violated, qGREP runs a risk of producing bad solution. To avoid this, some mechanism should be introduced, for example, adjusting the value of Q.

QoS-Guaranteed Ring Replication Management with Strong Consistency

43

3.3 A Simple Example of qGREP Here we present an example of qGREP to show its workflow as well as verify its correctness. A distance matrix DM can be generated from a storage graph illustrated by Fig. 1. Since the graph has bidirectional links, the matrix must be symmetric.

DM

⎡ 0 ⎢ 3 ⎢ ⎢ 7 ⎢ ⎢ 28 ⎢ 14 ⎢ ⎢ 5 ⎢ 9 ⎢ = ⎢ 19 ⎢ 23 ⎢ ⎢ 19 ⎢ ⎢ 16 ⎢ 11 ⎢ ⎢ 14 ⎢ 22 ⎢ ⎣⎢ 20

＝

3 0

7 9

28 31

14 17

5 8

9 12

19 22

23 28

19 22

16 19

11 14

14 17

22 25

9 31 17

0 35 21

35 0 42

21 42 0

12 33 19

16 19 23

26 29 33

32 48 39

26 42 33

23 36 38

18 31 25

21 34 28

29 42 36

8 12 22

12 16 26

33 19 29

19 23 33

0 14 24

14 0 10

24 10 0

20 29 39

14 23 33

11 17 27

6 12 22

9 15 25

17 23 33

28 22 19

32 26 23

48 42 36

39 33 38

20 14 11

29 23 17

39 33 27

0 6 11

6 0 5

11 5 0

17 11 5

14 8 2

28 22 16

14 17 25

18 21 29

31 34 42

25 28 36

6 9 17

12 15 23

22 25 33

17 14 28

11 8 22

5 2 16

0 3 11

3 0 14

11 14 0

23

27

40

34

15

21

31

20

14

8

9

6

20

20 ⎤ 23 ⎥⎥ 27 ⎥ ⎥ 40 ⎥ 34 ⎥ ⎥ 15 ⎥ 21 ⎥ ⎥ 31 ⎥ 20 ⎥ ⎥ 14 ⎥ ⎥ 8 ⎥ 9 ⎥ ⎥ 6 ⎥ 20 ⎥ ⎥ 0 ⎦⎥

Q=16, ξ 0.2, R={0}, AV=DM[0]= [0 3 7 28 14 5 9 19 23 19 16 11 14 22 20] , D= MAX(DM[ • , • ])=48, Q>48*0.2/0.8=12, so this graph has a feasible solution. The workflow looks like this: 1) Since AV[4]=28>Q, qGREP enter the loop, calculating PV of each row and then choose node 10 obtaining the biggest Profit (79) to join into the replica ring, thus R={0, 10}. After the first loop, AV=[0 3 7 28 14 5 9 19 11 5 0 5 2 16 8] 2) Since AV[4]=28>Q again, qGREP calculates PV of each row and then choose node 3 obtaining the biggest Profit (28) to join into the replica ring, thus R={0, 3, 10}. After the second loop, AV=[0 3 7 0 14 5 9 19 11 5 0 5 2 16 8] . 3) Since AV[8]=19>Q, qGREP calculates PV of each row and then choose node 8 obtaining the biggest Profit (19) to join into the replica ring ,thus R={0, 3, 10, 8}. Because d(0,8)+d(8,3) > d(3,8)+d(8,10) > d(10,8)+d(8,0), 8 is inserted between 0 and 10. After the third loop, AV=[0 3 7 0 14 5 9 0 11 5 0 5 2 16 8] . 4) Now all elements in AV are less than Q, so the algorithm ends with replicas at node of 0, 3, 8 and 10.

4 Ring-Based Election and Consistency Maintaining In our model both query and update requests can be forwarded to any replica node, thus achieving a good balance of overload. When several updates arrive at different sites simultaneously, good consistency mechanism is in urgent need to handle the conflict problem. We use a unidirectional ring to run election algorithm. Winner updates its replica and propagates it to all the other replica sites, but losers have to wait for the next round of election. Obviously, our model behaves a strong consistency to data requests. A unidirectional ring structure is a graph that in-degree and out-degree of all nodes are both 1, as depicted in Fig. 1. Many graceful ring-based electing algorithms can be

44

W. Fu, N. Xiao, and X. Lu

employed here. We present a special vote-alike algorithm called VOSE. Every node maintains a list of update initiators named elector. Each update initiator sends out a message vote to announce himself as an elector, and other nodes are electorates. Once receiving a vote, a node adds it to the list. Then it chooses a node from the list and sends its id to its next neighbor. Once an elector receives a vote including his own id, it wins the elections and the election terminates. A pseudo-code is illustrated as Fig. 3 Var: Listp; Begin 1 If p is an elector, then 2 Add p into Listp, status = wait, send; 3 Receive; 4 If p≠q, then 5 Listp=Listp [1], let r=GetMin(Listp), send ; 6 Else //p=q 7 p wins, the election terminates.

∪

8

//otherwise, p must be an electorate

9 Else looping until the election terminates 10 Receive , add p into Listp; 11 let r=GetMin(Listp), send ; //r has a minimal cost End Fig. 3. Distributed algorithm: VOte Swallow Electing (VOSE)

Line 5/11:

∈

∀ u List, let Pro(u)= MAX (d (u, v)) , GetMin(List) chooses the id of some v∈R

node who makes the minimal Propagating cost Pro(u). If several nodes have the same value of Pro(u), the one with smallest label will be selected. Since every node has information about the topology and DM, this can be done within (n2). After a round of election, the winner terminates the election and performs updating using broadcasting. From PR2 every node knows the position of other replica nodes and communication cost between them, so broadcasting can be employed to propagate the update. Furthermore, the cost should be the largest delay between winner and any node in the replica ring. As illustrated in Fig. 1.B, propagating cost of node 0 is Pro(0)=max{10, 23, 28}=28. Since GetMin(List) selects an elector with minimal Pro(u), the update cost is minimized. Note that the broadcasting way is not constricted along the ring. Votes from electors with smaller Pro(u) value always “swallow” those with bigger value, hence it is named VOSE. It has a message complexity of (n2) in the worst cases, and time complexity is (n2). Additionally, G, DM and Listp requires a cost complexity of (n2). Detail information please refer to [15].

Θ

Θ

Θ

Θ

5 Experiments and Evaluation Methodology of our Experiments In order to simulator the real internet environment, we adopt the famous network topology generator BRITE to produce networks with different size. The information

QoS-Guaranteed Ring Replication Management with Strong Consistency

45

about interconnecting and delays are extracted from output.brite files to build up adjacency matrix AM. Then the Floyd-Warshall algorithm is employed to generate distance matrix DM from AM, with time complexityΘ(n3). This cost can be awful when n is big enough. Fortunately, in Grid system, nodes are often made up of trustful servers with powerful capacity and strong resistibility to failure, as declared in PR1. So the effectiveness of such a DM can last a long time. Once RDG notices that the topology’s changing, it collects new information and generates DM again. In BRITE, firstly N nodes are placed into a square plane ordered by HS and LS, either randomly or referring to the heavy tail phenomenon. Then edges are added to interconnect nodes with a probability p. Before outputting, attributes such as delay and bandwidth are assigned to topology components. In our experiments, HS=10*N, LS=N and p= 0.45. In our experiments, the effects of QoS restrict Q is replaced by a scale of QoS restrict and network diameter D, i.e., Q=QoS/D for the sake of practicability. Every experiment repeats 10 times with random selected original site. Query cost T is substituted by D*rn/n, where rn is the replica number, and n is the total number. Thus total access cost C equals to (1-ξ)* Q + ξ*T. Q 

Q 

Q 

Q 

Q 

Q 

Q 









DFFHVVGHOD\VHFRQG

1RRIUHSOLFDV

Q           

     



    SURSRUWLRQRIXSGDWH¯









SURSRUWLRQRIXSGDWH¯

Fig. 4, 5. Given Q=D*0.1, with the growth of nodes, both replica number and access cost keep stable as the proportion of update increases Q 

Q 

Q 

Q 

Q 

Q 

Q 

WLPHFRVWVHFRQG



1RRIUHSOLFDV

Q 

          



      



  VFDOHRI4'









  VFDOHRI4'





Fig. 6, 7. Given ξ=0.1, n and C decrease fast with the expanding of Q/D

Above all, we study how the access pattern, i.e. proportion of update ξ, impacts the system performance and the replica placement. The QoS Q is set to be D/10. Results are illustrated in Fig. 4 and 5. It is obvious that rn and C both keep steady even if n increases a lot. There are two explanations for this phenomenon. Firstly, we suppose that query happens much more frequently than update, so the weight of update ξ is

46

W. Fu, N. Xiao, and X. Lu

relatively small. Secondly, qGREP gains enough query benefits to counteract increment of costly updating operations. Next we exchange the role of Q andξ. Whenξis fixed as 0.1, results can be depicted as Fig. 6 and 7. Totally different from Fig. 4 and 5, Fig. 6 and 7 declare that Q is one of the most important factors in RDG model. If the QoS restrict is relaxed below about 1/3 of the network diameter D, qGREP only needs a small number of replica nodes for better performance as well as less storage consumption. Furthermore, Fig. 8 illustrates the relationship between replica percentages and Q. It shows that network with more nodes generate less replica percentages. It is make sense because in BRITE generated topology, as number of nodes increases, the number of edges increases correspondingly. Thus a replica node has more opportunities to serve other non-replica nodes in his vicinity. As a result, a comparatively smaller replica percentage can fulfill the QoS restrict.

Fig. 8. Another profile of the same experiments, except Y axis’s substituting by a percentage

6 Related Work Our work has close relationship with replica management, QoS, and replica consistency problem. We will talk about them respectively. Replica Placement and QoS Problem Replica Placement and QoS problem have both been widely studies in grid researches[12, 13, 16]. Almost all of the replica placement pay more attention to optimize the overall performance, such as the mean access latency or TTL hops. They seldom think about each part individually, such as QoS requirement. Literature [12] associates replica placement with workload. It selects optimal locations for replica so that the workload of each node becomes balance. Recently some inspirations emerged to meet the individual requirement. Literature [14] proposed QoS-Aware Replica Placement Problem for replica-aware service and

QoS-Guaranteed Ring Replication Management with Strong Consistency

47

replica-blind service. It promises that any quest can be served within a given QoS requirement, and the replica-aware problem is proved to be NP-complete. Two heuristic algorithms named l-greedy-insert and l-greedy-delete is provided for optimal solution. On the basis of the work, [17] gave another proof of NP-hard property of this problem as well as two distributed algorithms. The newest research [13] proposes another heuristic algorithm that determines the positions of replicas with minimized costs. All of them only care about query for read-only objects. Instead, we study the placement policy considering both query cost and update cost. While QoS of query is similar to [13], we turn the QoS of overall cost to be a question under the concept of probability, associating with a proportion of update. We believe this has never been carefully studied before. Ring-based Replica Consistency Problem Most of the literatures[2, 3, 11] about Data Grid always regard data as read-only objects. Due to the consideration of data update, consistency becomes an unavoidable problem in our model. Literature[18] gave some models for replica synchronization and consistency in Data Grid and classified 5 level of Consistency Levels. However, it presented no practical solutions. As for the structure of replica, Literature[5] presented a chain-based replication for high throughput and availability while maintaining strong consistency in cluster computing. In his model, every query request is directed to the tail, while every update request is directed to the head of the chain. This brings problem of overload and hot accessing. Furthermore, the update cannot be initiated at multiple nodes simultaneously. Additionally, Literature [19] gave a tree structure of replica management, which is based on the tree-like structure of the system it concerns about. In our ring-based Data Grid model, every node can make response for query or update request, which achieves good balance of overload. Ring-based electing algorithm is presented to negotiate update conflicting.

7 Conclusion and Future Work Achieving strong consistency produces considerable additional overhead for distributed system, especially Data Grid with multiple writable data objects. In this paper, we exploit the management capability of Grid system and propose a novel ring-based replica consistency maintaining mechanism to negotiate update conflicting, supporting simultaneous query & update access to any replica site. Comprehensive consideration of individual QoS requirement has not been emphasized before. We present a QoS-guaranteed replica placement algorithm to make an ideal trade-off between query cost decrement and update cost increment while promising each user can obtain what he want in a pre-defined time limit. A qualification inequation reveals the internal relationship between the diameter, QoS and access pattern. It must be pointed out that our algorithms depend on the functionality of Grid heavily. The topology and distance information should be collected and delivered to every participant. If the underlying network changes all the time, the additional cost of DM reproducing will become unacceptable. Furthermore, data update frequency,

48

W. Fu, N. Xiao, and X. Lu

represented by the proportion of update ξ, should not be very frequent. From the Qualification inequation Q>ξ* D/(1-ξ), if ξ = 0.2, then Q>D/4; if ξ = 0.8, then Q >4D. This can be intolerant in Data Grid where D is very big. Till now, we discuss the problem in a static network. In the future we will study the influence of node adding/quitting, as well as the failure-tolerance of replica nodes. We believe that a lazy algorithm should be developed to obtain the DM efficiently. Also relatively relax consistency model for RDG is under consideration.

References 1. Wei Fu, Q.H.: GridEye: A Service-oriented Grid Monitoring System with Improved Forecasting Algorithm. In: The 5th International Conference on Grid and Cooperative Computing, Changsha, China (2006) 2. Ann Chervenak, I.F., Kesselman, C., Salisbury, C., Tuecke, S.: The Data Grid: Towards an Architecture for the Distributed Management and Analysis of Large Scientic Datasets. Journal of Network and Computer Applications 23, 187–200 (2001) 3. Hoschek, W., F.J.J.-M., Samar, A., Stockinger, H., Stockinger, K.: Data Management in an International Data Grid Project. In: Proceedings of the 1st IEEE/ACM International Workshop on Grid Computing. 2000, Bangalore, India, Springer-Verlag, Heidelberg (2000) 4. Dahlin, M.D., R.Y.W., Anderson, T.E., Patterson, D.A.: Cooperative Caching: Using Remote Client Memory to Improve File System Performance. In: Proceedings of Operating Systems Design and Implementation (OSDI’94) (1994) 5. Schneider, R.v.R.a.F.B.: Chain Replication for Supporting High Throughput and Availability. In: 6th Symposium on Operating Systems Design and Implementation(OSDI’04) (2004) 6. Foster, L.W.D.a.D.V.: Comparative models of the file assignment problem. ACM Computing Surveys 14(2), 287–313 (1982) 7. Satyanarayanan, M.: Coda: A Highly Available File System for a Distributed Workstation Environment. IEEE Trans. on Computers 39(4), 447–459 (1990) 8. Jia, X., D.L., Hu, X.-D., Du, D.-Z.: Placement of read-write web proxies in the internet. In: ICDCS’01 (2001) 9. Jianliang, X., B.L., Lee, D.L.: Placement Problems for Transparent Data Replication Proxy Services. IEEE JOURNAL ON SELECTED AREAS IN COMMUNICATIONS 20(7), 1383–1398 (2002) 10. David, W.B., D.G.C., Capozza, L., Millar, A.P., Stocklinger, K., Zini, F.: Simulation of dynamic grid rdeplication strategies in optorsim. In: Proceedings of 3rd Intl IEEE Workshop on Grid Computing (Grid’02) (2002) 11. Stockinger, H., A.S., Allcock, B., Foster, I., Holtman, K., Tierney, B.: File and object replication in data grids. In: 10th IEEE Symposium on High Performance and Distributed Computing (HPDC’01) (2001) 12. Pangfeng, L., J.-J.W.: Optimal Replica Placement Strategy for Hierarchical Data Grid Systems. In: Proceedings of the Sixth IEEE International Symposium on Cluster Computing and the Grid (CCGRID’06) (2006) 13. Hsiangkai, W., P.L., Jan-Jan, W.: A QoS-Aware Heuristic Algorithm for Replica Placement. in Grid Computing Conference 2006 (2006) 14. Xu, X.T.a.J.: Qos-aware replica placement for content distribution. IEEE TRANSACTIONS ON PARALLEL AND DISTRIBUTED SYSTEMS 16(10), 921–932 (2005)

QoS-Guaranteed Ring Replication Management with Strong Consistency

49

15. Tel, G.: Intorduction to Distributed Algorithms, 2nd edn. Publishing House of Electronics Industry. pp. 227–267 (2003) 16. Kavitha Ranganathan, I.F.: Identifying Dynamic Replication Strategies for a HighPerformance Data Grid. In: Grid Computing Conference 2001 (2001) 17. Jeon, W.J., I.G., Nahrstedt, K.: QoS-aware Object Replication in Overlay Networks. In: IPTPS 2005 (2005) 18. D”ullmann, D., W.H., Jaen-Martinez, J., Segal, B., et al.: Models for Replica Synchronisation and Consistency in a Data Grid. In: IEEE Symposium on High Performance Distributed Computing HPDC’, 2001. Redondo Beach, California, USA (2001) 19. Kalpakis, K., K.D., Wolfson, O.: Optimal Placement of Replicas in Trees with Write, and Storage Costs. IEEE TRANSACTIONS ON PARALLEL AND DISTRIBUTED SYSTEMS 12(6), 628–637 (2001)

A Mobile Database Sharing Protocol to Increase Data Availability in Mobile Environments Hien Nam Le1 and Mads Nygård2 1 Department of Telematics [email protected] 2 Department of Computer and Information Science, Norwegian University of Science and Technology (NTNU) [email protected]

Abstract. This paper presents a mobile database sharing protocol to improve data availability in mobile environments. Long disconnection periods of mobile hosts could cause data inconsistency and blocking of mobile transactions. The proposed mobile database sharing protocol allows two conflict data caching modes to reduce the blocking of mobile transactions that access to shared data. Furthermore, data can be directly shared among mobile hosts that are being disconnected from the database servers. Data consistency is ensured via the execution constraints and dependency among transactions. Keywords: Mobile database, mobile data management, conflicting operations.

1 Introduction The rapid growth of wireless network technologies and capacity of portable computing devices has promoted a new mobile environment. People can access to information and carry out their transactions while being on the move. The environment for accessing and processing information is changing from stationary to mobile and location independent. A mobile environment is characterized by unique characteristics: the mobility of users or computers, the frequent and unpredictable disconnection of wireless networks, and the limitation of computing capacity of mobile devices [1]. These characteristics pose many challenges for mobile transaction processing. Long disconnection periods in communication could cause data inconsistency or blocking of other transactions. For example, if a data item is locked at a disconnected mobile computer for a long period, this could delay the progress of transactions on other mobile hosts. The duration of disconnected or connected periods is not always as planned, i.e., varying in time, due to many factors, for example, the limited batterylife of mobile computers, the unstable bandwidth of wireless networks. This might result in longer delay time or interruption periods for database operations. In this paper, we present a mobile database sharing protocol to increase data availability in mobile environments. The main contributions of our research include: (1) two conflict caching modes to allow accessing to shared data objects that have been locked by mobile hosts that are being disconnected from the database servers for K.C. Chang et al. (Eds.): APWeb/WAIM 2007 Ws, LNCS 4537, pp. 50–61, 2007. © Springer-Verlag Berlin Heidelberg 2007

A Mobile Database Sharing Protocol to Increase Data Availability

51

long periods; and (2) mobile data sharing protocols to support direct data sharing among mobile hosts that are being disconnected from the database servers. We consider a large database DB that is distributed among several fixed servers Si. The correctness criterion of the system is the serializable schedule of transactions [4]. Data items are cached at mobile hosts for disconnected transaction processing. To avoid data inconsistency among transactions at different mobile hosts, the database servers grant read or write locks on data items that are being downloaded into the mobile hosts (before that mobile hosts are disconnected from the database servers). When mobile host MHi is being disconnected from the database servers, it can share data with other mobile hosts MHj via the support of mobile affiliation workgroups (reviewed in the next section). We also distinguish two classes of transactions in mobile environments: online transaction and offline transaction. An online transaction is a transaction that interacts directly with the transaction manager at the fixed database servers to perform read or write operations on shared data. An offline transaction is a transaction that is executed and managed by the mobile transaction manager at the disconnected mobile host. Offline transactions are first locally committed at the disconnected mobile host, and then finally be verified and committed at the database servers. A locally committed offline transaction may be finally aborted if there is a non-serializable execution occurred. We use the following notations: Tik denotes a transaction Tk at mobile host MHi. A transaction Tik that shares data to or obtains data from other transactions is called a delegator or delegatee transaction, respectively. In some cases, a transaction can play roles as both delegator and delegatee transaction. The read and write locks on shared data item X are denoted by XR and XW, respectively; RX and WX represent the read and write operations on the shared data item X. The rest of the paper is organized as follows. Section 2 discusses the related research. Section 3 presents the conflict caching modes. The mobile database sharing protocols is addressed in Section 4, and Section 5 concludes the paper.

2 Related Research In this section, first we briefly recap the concepts of mobile affiliation workgroups and mobile data sharing workspace that support data sharing among mobile hosts. Later, we review several related work. 2.1 Mobile Affiliation Workgroup Model and Mobile Sharing Workspace Our mobile database sharing protocol is developed based on the concepts of mobile affiliation workgroups and mobile data sharing workspace (as illustrated in Figure 1). The objective of the mobile affiliation workgroup model is to support information sharing among transactions at different disconnected mobile hosts in mobile environments. This can be done by allowing such disconnected mobile hosts to form dynamic mobile affiliation workgroups (MAW) [11] by taking advantages of wireless communication technologies, i.e., the ability of direct communication among mobile hosts within a limited area. Sharing of information within a mobile affiliation workgroup is carried out through a dynamic mobile sharing workspace created by participating mobile hosts. Further detail about the mobile affiliation workgroup can be found in [12].

52

H.N. Le and M. Nygård

3

5

EI1

EI2

1

4

MA1

2

Disconnected mobile host

Wireless LAN

Connected mobile host

Short-range wireless

Mobile support station

High-speed wired networks

Mobile sharing space

Mobile affiliation (MA)

MA 2

DBMS

Fig. 1. Mobile affiliation workgroup and mobile sharing workspaces

2.2 Related Research Mobile transactions and mobile databases have been wisely applied to cope with the challenges of sharing data among mobile transactions. However, there are still some major limitations. The architecture of mobile transaction environments [5] relies too much on mobile support stations that are stationary and connected. Current approaches that tackle long locking periods among transactions may not be suitable in mobile environments. For example, altruistic locking protocols [9] could cause cascading aborts; speculative locking protocols [8] might require extra computing resources at mobile hosts. The delegation approach [7,10] that supports sharing information among transactions may not be adequate because it requires tight cooperation, i.e., connectivity, among delegator and delegatee transactions. [6] presents a multi-version transaction model for improving data availability in mobile environments. A mobile transaction in this model has three different states of execution: start, commit and termination. The shared data updated by a transaction Ti is made available to another transaction Tj immediately after Ti commits. In this model, however, a data item could be locked for long periods of time at a mobile host until the mobile host releases that locks at mobile support stations. In [3], four different disconnection modes for mobile databases are presented: basic sign-off, check-out, relaxed check-out and optimistic check-out. This model only supports planned disconnected database operations for one mobile host, while our mobile data sharing protocol can deal with both planned and unplanned disconnections of a single host or a group of mobile hosts. To support dynamic database sharing in a mobile database community, the Accessing Mobile Database architecture [2] based on the concept of mobile agents is proposed. The limitation of this architecture is that the role of a mobile host is restrained to either the database server or a database client. Our mobile data sharing mechanism does not discriminate between a mobile database server and a mobile database client.

3 Conflict Caching Modes In this section, we present the two conflict caching modes that play important roles in the mobile database sharing protocols. According to [4], two database operations Opi and Opj of two transactions Ti and Tj are in conflict if they are: (1) accessing the same data item, (2) one of them is a write operation. The conflict of database operations is denoted by Conflict(Opi,Opj).

A Mobile Database Sharing Protocol to Increase Data Availability

53

Definition (directly conflicting transactions). Two transactions Ti and Tj are in direct conflict, denoted by T i C d T j , if there is an operation Opi of transaction Ti that conflicts with an operation Opj of transaction Tj. Definition (indirectly conflicting transactions). Two transactions Ti and Tj are in indirect conflict, denoted by T i Cid T j , if there is a transaction Tk that Ti either develops a direct conflict or an indirect conflict with, and Tk develops either a direct conflict or an indirect conflict with Tj, i.e.,

T i Cid T j if ∃T k , (T i C d T k ∨ T i C id T k ) ∧ (T k C d T j ∨ T k C id T j ) If the database servers will make use of the standard compatibility between read and write locks to ensure data consistency, there is no conflict lock request is allowed. However, in mobile environments, this seems to be too restricted to be useful, due to, long disconnection periods of mobile hosts. In our mobile transaction processing system, there are two different data caching modes: read-write conflict and write-read conflict. These conflict mobile data caching modes are discussed below. 3.1 Read-Write Conflict Data Caching Mode

Fig. 2 illustrates the read-write caching conflict scenario. Suppose that an online transaction T1i at connected mobile host MH1 is holding a read lock XR on a shared data item X, and an offline transaction T2j at mobile host MH2 requests a write lock XW on this shared data. The write lock request can be granted to the offline transaction T2j because the write operation WX by transaction T2j is not immediately carried out at the database servers, even after the online transaction T1i has committed. And the transaction T1i is scheduled to execute before transaction T2j, i.e., T1i→ T2j (the execution constraints are discussed in next section). To handle this limitation, the mobile transaction processing system will allow these conflict lock requests to be compatible.

Fig. 2. Read-write conflict data caching mode

Definition (read-write conflict). If an online transaction Tik holds a read lock on data item X, and an offline transaction Tjl requests a write lock on data item X, the database server grants the write lock to Tjl. We call this conflict mode a read-write (RW) conflict and denote it XRW(Tik, Tjl).

Our read-write conflict mode focuses on supporting offline transactions at the disconnected mobile hosts. The read-write conflict provides the mobile transaction processing system the ability to avoid blocking of the execution of an offline updating

54

H.N. Le and M. Nygård

transaction, i.e., if the shared data item X is read locked by an online transaction Tik, the write lock request from an offline transaction Tjl will be granted. In Figure 2, when the mobile host MH2 reconnects to the database servers, the write (offline) transaction T2j will be converted to an online transaction (i.e., with online write lock on the shared data item X) so that the updated data value VX’ will be integrated into the database servers. At this time, any on-going online transaction Tp that currently holds read lock on the shared data item X is either allowed to commit (given that the final commitment of the transaction T2j will be delayed) or aborted. We define the execution constraint among transactions that access shared data in conflict modes as follows: Definition (execution constraint). A transaction Ti is said to be scheduled before a transaction Tj, denoted by Ti → Tj, if all the conflicting operations Opi of transaction Ti is executed before the conflicting operations Opj of transaction Tj , i.e.,

Ti → Tj ⇔ (∀Opi∈ Ti, Opj∈ Tj, Conflict(Opi,Opj) ⇒ Opi → Opj) Rule 1 (execution constraint of rw-conflict). If transaction Tik develops a read-write conflict with transaction Tjl on shared data X, i.e., transaction Tjl will modify the shared data X offline after it is being read by transaction Tik, transaction Tik will be scheduled before transaction Tjl, i.e., Tik → Tjl. 3.2 Write-Read Conflict Data Caching Mode

In read-write conflict data caching mode, a write lock request on the shared data item of an offline transaction is granted even if the shared data is currently being read lock by other transactions. On the other hand, an online transaction or an offline transaction can be allowed to read a shared data item while another offline transaction holds a write lock on the same shared data item, as long as these transactions can be serialized with the offline updating transaction. Figure 3 illustrates the write-read conflict scenario. The offline transaction T2j at disconnected mobile host MH2 holds a write lock XW on the shared data item X. However, this data item is not being immediately modified at the database servers because the mobile host MH2 that executes transaction T2j is currently being disconnected. When an (online or offline) transaction T1i at mobile host MH1 requests a read lock XR on the data item X, this read lock will conflict with the write lock on X held by transaction T2j. In this case, the database server can grant a read lock on X (and consequently allow the read operation to be executed) for transaction T1i, given the original value VX of the data item X is returned (this value might be inconsistent with the value of X that is stored and being modified at the disconnected mobile host MH2). In fact, at the database servers, the original data value VX is the most up-to-date and consistent data. Consequently, to ensure that the involved transactions are serializable, transaction T1i must be scheduled before transaction T2j, i.e., T1i → T2j. Note that the offline transaction T2j may not know about this conflict that is happening at the database servers. To handle this limitation, the mobile transaction processing system will allow these conflict lock requests to be compatible.

A Mobile Database Sharing Protocol to Increase Data Availability

55

Fig. 3. Write-read conflict data caching mode

Definition (write-read conflict). If an offline transaction Tik holds a write lock on data item X, and a transaction Tjl requests a read lock on data item X, the database server grants the read lock request and the un-modified value of X is returned. We call this conflict mode a write-read (WR) conflict and denote it XWR (Tik,Tjl).

The write-read conflict mode allows read operations to be executed when there is a write operation that is being executed at the disconnected mobile host, i.e., avoids blocking of the execution of the read operations on the shared data item. In Figure 3, when the mobile host MH2 reconnects to the database servers, the write (offline) transaction T2j will be converted to an online updating transaction with an online write lock on the shared data item X so that the updated data value VX’ will be integrated into the database servers. At this time, any on-going online transaction Tp that currently holds a read lock on the shared data item X is either allowed to commit (given that the final commitment of the transaction T2j will be delayed) or aborted. Rule 2 (execution constraint of wr-conflict). If transaction Tik develops a write-read conflict with transaction Tjl on shared data X, i.e., transaction Tik will read the shared data X after it is being modified offline by transaction Tjl, transaction Tik will be scheduled before transaction Tjl, i.e., Tik → Tjl.

4 Mobile Database Sharing Protocol In this section, we present the mobile database sharing protocols that support sharing of information among transactions at different mobile hosts. Mobile hosts will cache shared data item before disconnecting from the database servers. While being disconnected from the database servers, the mobile hosts can directly shared its cached data to other hosts. For a shared data item, it can be cached either with a read lock or a write lock at a mobile host. If a shared data item X is cached with read lock XR at the mobile host MHi, an offline transaction Tik can only perform read operation RX on X. On the other hand, if X is cached with a write lock XW, the offline transaction Tik can modify X, i.e., write operation WX. Hence, depending on a status of a cached data item, a delegator transaction Tik can shared either an original shared data or an updated shared data to a delegatee transaction Tjl (Table 1 summaries these sharing data options). If a delegator transaction Tik at mobile host MHi holds a read lock on a data item X, the shared data value VX will be identical to the value cached at the mobile host, i.e., the original data state is shared. However, if the delegator Tik at mobile host MHi holds a write lock on

56

H.N. Le and M. Nygård

data item X, the shared value VX can be either an old value VX (i.e., before the delegator transaction updates X) or an updated value VX’ (i.e., after the delegator transaction has updated X). Table 1. Locks and equivalent shared data states

Shared data

Original value VX Modified value VX’

Lock on X Read Write Relevant Relevant N/A Relevant

Delegatee transactions Tjl at other mobile hosts MHj are only allowed to read these shared values, i.e., to the target mobile hosts, the shared data item are read-only. By this way, a local transaction at the disconnected mobile host can obtain the needed data without being connected to the database servers, i.e., reducing the blocking period. This is one of the novel advantages of our mobile data sharing mechanism to increase the data availability in mobile environments. 4.1 Sharing Original Data

In this sub-section, we discuss the execution constraints when a delegator transaction k l Ti shares an original cached data item to a delegatee transaction Tj . The following rule defines the execution constraint between these mobile transactions: Rule 3 (execution constraint of sharing original data state). If delegator transaction Tik shares an original data state to delegatee transaction Tjl, transaction Tjl must be scheduled before transaction Tik, i.e., Tjl → Tik.

This rule describes the mobile sharing data states scenario in which a delegator transaction Tik shares an original data state VX of the data item X to a delegatee transaction Tjl. The delegator transaction Tik can hold a read lock, or a write lock on the shared data item but the shared data state has not been modified. In this scenario, both the delegator Tik and delegatee Tjl transactions read the same value VX of data item X. If the delegator transaction Tik reads a consistent data value of X, then the delegatee transaction Tjl will be assured to read the same consistent data value as the delegator transaction Tik. If the delegator transaction Tik holds a read lock XR on X and there is another transaction Txy (at a different mobile host) with which the delegator transaction Tik holds a read-write conflict or a write-read conflict, i.e., Tik → Txy, this rule ensures that Tjl →Tik → Txy, i.e., both transactions Tik and Tjl read consistent data values in relation to the transaction Txy. If the delegator transaction Tik holds a write lock XW on X, and there is another transaction Txy (at a different mobile host) with which the delegator transaction Tik holds a read-write conflict or a write-read conflict, i.e., Txy → Tik, this rule ensures that either Tjl → Txy → Tik or Txy →Tjl → Tik, i.e., both transactions Tjl and Txy read consistent data values in relation to the transaction Tik.

A Mobile Database Sharing Protocol to Increase Data Availability

57

In Figure 4(a), an example of sharing original values with a read lock is shown. Time proceeds from left to right. The delegator transaction T11 at the mobile host MH1 holds a read lock on the shared data item X and shares the value VX to the delegatee transaction T21 at the mobile host MH2. If these two transactions T11 and T21 finally commit when the mobile hosts reconnect to the database servers, the transaction T21 must be scheduled before the transaction T11, i.e., T21→ T11. In Figure 4(b), an example of sharing original values with a write lock is shown. Time proceeds from left to right. The delegator transaction T11 at the mobile host MH1 holds a write lock on data item Y and shares the original (i.e., non-modified) value VY to the delegatee transaction T21 at the mobile host MH2. In this case, the final transaction schedule will again be T21→ T11.

(a)

(b)

Fig. 4. Execution constraints with sharing original data

4.2 Sharing Modified Data

When a delegator transaction shares an updated data state to a delegatee transaction, the following rule is applied: Rule 4 (execution constraint of sharing updated data state). If delegator transaction Tik shares an updated data state to delegatee transaction Tjl, transaction Tjl must be scheduled after transaction Tik and before any transaction Tin that is scheduled - due to another update - after transaction Tik in the locally committed transaction set LCi at the same mobile host, i.e., ∀Tin ∈ LCi, Tik ∈ LCi , Tik < Tin ⇒ Tik → Tjl → Tin This rule is denoted by Tik →• Tjl.

This mobile sharing data scenario happens when a delegator transaction Tik holds a write lock XW on the shared data item X, and the shared data item has been modified. If the delegatee transaction Tjl were only to be scheduled after the delegator transaction Tik, and the shared data item is later modified again by another transaction Tin (the transaction Tin is executed at the same mobile host as the delegator transaction and also scheduled after Tik), the execution schedule Tik→Tin→Tjl will not be correct. Instead, the correct execution schedule must be Tik→Tjl→Tin, i.e., with the above rule Tik→•Tjl being met. In Figure 5, an example of sharing updated values with a write lock is shown. The transaction T11 shares the new value VY’ to the delegatee transaction T21. At mobile host MH1, there is another transaction T12 that later updates it to a new value VY’’. The transaction T12 is scheduled after the transaction T11. Rule 4 ensures that the

58

H.N. Le and M. Nygård

transaction T21 will be scheduled between transactions T11 and T12. This means that the final global transaction schedule is T11→T21→T12. When the mobile hosts MH1 and MH2 reconnect to the database servers, this transaction execution constraint will be used to support the transaction integration process.

Fig. 5. Execution constraints with sharing modified data

Rule 5 (transaction dependency). If a delegator transaction Tik shares a modified data item X to a delegatee transaction Tjl , the delegator transaction Tik will hold an abort-dependency with transaction Tjl on shared data X, i.e., if the transaction Tik aborts, then the transaction Tjl aborts.

The above rule will ensure that if an offline transaction is operating on an invalid shared data item, this offline transaction will eventually abort when it reaches transaction integration stage (explained in the next section). 4.3 Transaction Integration

The transaction integration stage is carried out when the mobile host reconnects to the database servers. In this stage, locally committed transactions, which have been disconnectedly processing at the mobile host, will be validated against other transactions to ensure that the states of the database servers are consistent. In mobile environments, there is no guarantee that all the mobile hosts will synchronously connect to the database servers to integrate the locally committed transactions at the same time. Therefore, there is no guarantee that a delegator transaction will be integrated into the database servers before a delegatee transaction or via versa. Furthermore, a local transaction can play roles as both the delegator and delegatee transactions. Consequently, the database servers must keep track of the commit or abort state of both delegator and delegatee transactions in order to determine the effect of one transaction on the others (i.e., both the transaction execution constraints and dependencies must be checked). Figure 6 presents examples of these effects. In Figure 6(a), the delegator transaction T1i and the delegatee transaction T2j, which belong to different mobile hosts MH1 and MH2 respectively, develop an abort-dependency (T1i AD T2j) and a commit-dependency (T1i CD T2j). If the delegator transaction T1i commits or aborts before the delegatee transaction T2j, the final state of the delegatee transaction T2j can be determined normally. However, if the delegatee transaction T2j requests to finally commit before the delegator transaction T1i (as shown in Figure 6(b)), the final state of the delegatee transaction T2j will not be determined until the state of the delegator

A Mobile Database Sharing Protocol to Increase Data Availability

59

transaction T1i is known. In this case, the commit of the delegatee transaction T2j will be delayed, i.e., resulting in a pending commit.

Fig. 6. The effect of the order of transaction termination requests

The database servers will manage two transaction sets: (1) GlobalAbortedDelegator GAD and (2) PendingCommit PC. The GAD set will keep track of the abort of delegator transactions. If a delegator transaction is aborted, it will be added to the GAD set so that the associated delegatee transactions will be notified. For a delegatee transaction that holds an abort-dependency with a delegator transaction that has not been integrated yet, the commitment of the delegatee transaction will be delayed and added to the PC set. Handling the abort of delegator transactions The following algorithm takes care of the final aborts of delegator transactions TiDor and verifies the abort dependencies of associated delegatee transactions Tjl which are queued in the PendingCommit (PC) set. Dor

(1) For eachDordelegator transaction Ti that has aborted Add Ti to the GlobalAbortedDelegator set

(2)

l

For each Tj in the PendingCommit set that holds Dor an abort-dependency with transaction Ti l Abort Tj l If Tj is a delegator transaction l Add Tj to the GlobalAbortedDelegator set Fig. 7. Handling the abort and abort dependency of delegator transactions

The algorithm is presented in Figure 7 and is explained as follows: (1) Each of the aborted delegator transaction TiDor will be added to the GAD set. This will trigger a separate verification of the abort dependencies of the associated delegatee transactions. (2) Any associated delegatee transaction Tjl in the PC set holding an abortdependency with the delegator transaction TiDor will be aborted. If the aborted transaction Tjl is a delegator transaction (i.e., this transaction plays roles a both delegator and delegatee transactions), the transaction Tjl will be added to the GAD set.

60

H.N. Le and M. Nygård

Verify transaction dependency and execution constraint of offline transactions For an offline transaction Tik which has been locally committed at the mobile host, the offline transaction will be verified in two steps. First, any abort-dependency with a delegator transaction TjDor will be evaluated. Second, the execution constraints between transaction Tik with other transactions Tjl will be checked to ensure the correctness of transactions, i.e., serializable execution. Dor

(1) For eachDorabort-dependency with Tj If Tj is in kthe GlobalAbortedDelegator set Abort Ti k If Ti isk a delegator transaction Add Ti to the GlobalAbortedDelegator set Dor Else If Tj has not committed k Add Ti to the PendingCommit set l

(2) For all transactions Tj that hold execution k constraint with Ti Check the execution constraints l k l If Tj and Ti are non-serializable and Tj has committed Notify the transaction manager for manual handling l Else If Tj has notl committed m k Choose T = Tj | Ti m Abort T m If T ism a delegator transaction Addm T to the GlobalAbortedDelegator set k If T = Ti Exit the algorithm k Commit Ti Fig. 8. Verifying transaction dependencies of a locally committed transaction

The algorithm of this step is presented in Figure 7 and is explained as follows: (1) For each abort-dependency between the locally committed transaction Tik and a delegator transaction TjDor, if the delegator transaction TjDor has aborted, the transaction Tik must abort too. Otherwise, if the delegator transaction TjDor has not reached the transaction integration stage, the locally committed transaction Tik will be added to the PC set. In this case, the abort-dependency will be reevaluated when the termination state of the delegator transaction TjDor is known (see point (2) in Figure 7 above). (2) The execution constraints between transactions Tik and other transactions Tjl will be determined based on Rules 1-4. If transactions Tik and Tjl end up being nonserializable and Tjl has committed, a notification will be sent to transaction manager so that it can be handled separately, e.g., by compensating Tjl. If Tjl is alive, one of the transactions Tjl and Tik must be aborted. Then, we have to make a choice between it and Tik - which one depends on the policy to be used in a specific system. If the aborted transaction Tm is a delegator transaction, it will be added to the GAD set so that related pending transactions Tp in the PC set may

A Mobile Database Sharing Protocol to Increase Data Availability

61

be re-evaluated. After all the execution constraints are checked, transaction Tik will be finally committed at the database servers.

5 Conclusions In this paper, we present a mobile database sharing protocol to increase data availability in mobile environments. To deal with the problem of long disconnection periods, the mobile database sharing protocol support two conflict caching modes to allow accessing to shared data objects that have been conflicting locked by disconnected mobile hosts. Furthermore, the data sharing mechanism also support direct data sharing among mobile hosts that are being disconnected from the database servers. The data consistency is ensured via the execution constraints and dependency among delegator and delegatee transactions. Finally, a prototype of the mobile database sharing protocol has been successfully implemented [13]. Future work is to investigate the performance of the data sharing protocol.

References 1. Pitoura, E., Samaras, G.: Data Management for Mobile Computing. Kluwer Academic Publishers, Boston (1998) 2. Brayner, A., Filho, J.M.: Sharing Mobile Databases in Dynamically Configurable Environments, CAiSE 2003. In: Eder, J., Missikoff, M. (eds.) CAiSE 2003. LNCS, vol. 2681, pp. 724–737. Springer, Heidelberg (2003) 3. Holliday, J., Agrawal, D., Abbadi, A.E.: Disconnection Modes for Mobile Databases. Wireless Networks 8(4), 391–402 (2002) 4. Garcia-Molina, H., Ullman, J., Widom, J.: Database Systems: The Complete Book. Prentice Hall, Englewood Cliffs (2001) 5. Serrano-Alvarado, P., Roncancio, C., Adiba, M.E.: A Survey of Mobile Transactions. Distributed and Parallel Databases 16(2), 193–230 (2004) 6. Madria, S.K., Baseer, M., Bhowmick, S.S.: A Multi-version Transaction Model to Improve Data Availability in Mobile Computing, CoopIS/DOA/ODBASE, pp. 322–338 ( 2002) 7. Ramampiaro, H.: CAGISTrans: Adaptable Transactional Support for Cooperative Work, Dr.ing Thesis 2001:94, Norwegian University of Science and Technology (2001) 8. Reddy, P.K., Kitsuregawa, M.: Speculative Locking Protocols to Improve Performance for Distributed Database System. IEEE Transactions on Knowledge and Data. Engineering 16(2), 154–169 (2004) 9. Salem, K., Garcia-Molina, H., Shands, J.: Altruistic Locking. ACM Transactions on Database Systems 19(1), 117–165 (1994) 10. Chrysanthis, P.K., Ramamritham, K.: Synthesis of Extended Transaction Models Using ACTA. ACM Transactions on Database Systems 19(3), 450–491 (1994) 11. Le, H.N., Nygård, M.: A Mobile Affiliation Model for Supporting Collaborative Work in Mobile Environments, the 3rd Workshop on Ubiquitous Mobile Information and Collaboration Systems, pp. 649–660 ( 2005) 12. Le, H.N., Nygård, M.: Mobile Transaction System for Supporting Mobile Work, the 8th Workshop on Mobility in Databases and Distributed Systems, pp. 1090–1094 (2005) 13. Skjerven, S.: Adaptive Locking Protocols for Mobile Databases, Master Thesis, Dept. of Computer and Information Science, Norwegian University of Science and Technology (2006)

Mining Recent Frequent Itemsets over Data Streams with a Time-Sensitive Sliding Window Long Jin1, Duck Jin Chai1, Jun Wook Lee2,∗, and Keun Ho Ryu1 1

Database/Bioinformatics Laboratory, Chungbuk National University, Korea {kimlyong,djchai,khryu}@dblab.chungbuk.ac.kr 2 Telematics Research Division, ETRI, Daejeon, Korea [email protected]

Abstract. A data stream is a massive unbounded sequence of data elements continuously generated at a rapid rate. Consequently, the knowledge embedded in a data stream is more likely to be changed as time goes by. Mining frequent patterns is the one of them and has been widely studied over the last decade. There are several models and approaches, but there is only one study on the time-sensitive sliding window model. This study spends much memory and has a low accuracy. In this paper, we propose an efficient discounting method and a Sketch data structure for solving these problems. This approach has several advantages. (i) The accuracy is increased compared with that of previous techniques. The efficient discounting method not only loses the information about accumulated count but also decrease many missing true answers. (ii) The memory is saved. The Sketch data structure saves much space. (iii) It is not necessary to have the discount table and reduce significantly the computing time of discounting table. Experiment results show that our proposed method exactly increases the accuracy and saves the memory and the computing time.

1 Introduction A data stream is a massive unbounded sequence of data elements continuously generated at a rapid rate. Due to this reason, it is impossible to maintain all elements of a data stream. As a result, data stream processing should satisfy the following requirements [1]. First, each data element should be examined at most to analyze a data stream. Second, memory usage for analyzing data stream should be restricted finitely although new data elements are continuously generated in a data stream. Third, newly generated data elements should be processed as fast as possible. Finally, the up-todate analysis result of a data stream should be instantly available when requested. Frequent itemset mining is a core data mining operation and has been extensively studied over the last decade. There are two groups: finding frequent items and finding frequent itemsets on data stream. For finding these, there are four types of models. (i) Most of them [2, 3, 4] utilize all the data between a particular point of time (called landmark) and the current time for mining. The landmark usually refers to the time ∗

Corresponding author.

K.C. Chang et al. (Eds.): APWeb/WAIM 2007 Ws, LNCS 4537, pp. 62–73, 2007. © Springer-Verlag Berlin Heidelberg 2007

Mining Recent Frequent Itemsets over Data Streams

63

when the system starts. Moreover, the support count of an itemset in this model is the number of transactions containing it between the landmark and the current time. (ii) The landmark model is not aware of time and therefore cannot distinguish between new data and old ones. To overcome this difficulty, the time-fading model has been presented recent works [5, 6, 7]. It assigns different weights to transactions such that new ones have higher weights than old ones. All these approaches provide approximate answers for long-term data and adjust their storage requirement based on the available space. Therefore, they satisfy the two requirements – approximation and adjustability. (iii) In the sliding-window model, users can only be interested in the data recently arriving within a fixed time period. As a transaction arrives, the oldest transaction in the sliding window is expired. Therefore, under this model, the methods for finding the expired transaction and for discounting the support counts of the itemsets involved are required [8, 9, 10, 11]. (iv) In the time-sensitive sliding-window model [12], it is natural to specify a time period as the basic unit and regards a fixed time period as the basic unit for mining. A data stream is decomposed into a sequence of basic units, which are assigned with serial numbers starting at 1. This model slides the sliding window over this sequence to see a set of overlapping sequences. Lin et al. [12] proposed a time-sensitive sliding window model. They use the accumulated count (Acount) and the potential count (Pcount) concepts. Acount accumulates its exact support counts in the subsequent basic blocks. Pcount estimates the maximum possible sum of its support counts in the past basic blocks. There maintains a potential frequent itemsets pool (PFP) and deletes the itemset if the sum of its Acount and Pcount is less than the support count threshold ε. And they output the frequent itemsets only if these Acount is bigger than the threshold ε. There are many problems. First, the information about Acount is lost. There deletes the itemset c that doesn’t satisfy the threshold. The itemset c appears again at the after blocks. Although Acount is already computed, there is estimated Pcount and substituted Pcount for Acount. So the accuracy of Acount is decreased. Second, many true answers are missed. When filtering the frequent itemsets, they only use Acount. So there occur many missing true frequent itemsets by decreasing the accuracy of Acount. Also it needs to minimize the missing of true answers and guarantee that the no false answers are outputted, e.g. no-false-alarm. In this paper, we propose two kinds of solutions. First, we propose an efficient discounting method for increasing the accuracy of Acount. This efficient discounting method not only loses the information about Acount but also decrease many missing true answers. Second, we propose a Sketch data structure for decreasing the memory space. This structure saves much space than that of [12]. And it doesn’t need to have the discount table (DT) as in [12] and significantly reduces the computing time of DT. Third, we propose the detail of efficient algorithm. The remaining of this paper is organized as follows. Section 2 defines some symbols and definitions. The detail of efficient discounting method and Sketch data structure is shown in Section 3. Section 4 develops the detail of algorithm and shows the examples. Section 5 presents our performance of our proposed method. Section 6 summarizes this paper.

64

L. Jin et al.

2 Preliminaries Given a time point t and a time period p, the set of all the transactions arriving in [tp+1, t] will form a basic block. The number of basic block’s ciphers is |p|. A data stream is decomposed into a sequence of basic blocks, which are assigned with serial numbers starting at 1. Given a window with length |ω|, we slide it over this sequence to see a set of overlapping sequences, where each sequence is called the timesensitive sliding window abbreviated as TS. Let the basic block numbered i be denoted as Bi. The number of transactions in Bi is denoted as |Bi|, which is not fixed due to the variable data arrival rate. For each Bi, the TS that consists of the |ω| consecutive basic blocks from Bi-|ω|+1 to Bi is denoted as TSi. Let the number of transactions in TSi be denoted as Σi. For efficiently mining the frequent itemsets in TSi, we design the data structure that is consisted of one SA and one PFP as follows. 1.

SA is abbreviated the Support Array. The basic blocks may have different numbers of transactions and we store it into an entry in the SA. In there, only |ω|+1 entries are maintained in SA. As Bi arrives, SA[j] keeps the support count of Bi-j for 1 j |ω|+1 and i-j>0. After Bi is processed, the last entry SA[|ω|+1] is ignored and the others are moved to the next positions, i.e., SA[j]ÆSA[j+1] for 1 j |ω|. Finally, the support count of Bi is put into SA[1]. PFP is abbreviated the potential frequent itemsets pool. PFP includes the itemsets that are frequent in Bi but not frequent in TSi-1 in PFP because they are possibly frequent in TSi. PFP is consisted of an ID, an itemset, an accumulated count Acount, and a potential count Pcount. Where, Acount is the type of Sketch data structure that will describes in Section 3.2. This kind of Acount stores all accumulated counts from Bi-|ω|+1 to Bi. So there doesn’t need the discounting table (DT) in [12].

≤≤

2.

≤≤

This structure efficiently stores the data and minimizes the memory space. We will describe how to efficiently store the data and minimize the memory space in Section 3.

3 Our Proposed Method This section shows how to increase the accuracy and save the memory. For these ones, we propose an efficient discounting method in Section 3.1 and a Sketch data structure in Section 3.2. 3.1 Efficient Discounting Method For increasing the accuracy of Acount, we use a error rate r and control the accuracy of Acount. For deleting the infrequent itemsets and maintaining the potential frequent itemsets, there are two kinds of deletions. First is the logical deletion that the sum of Acount and Pcount is less than the support count threshold ε. The logical deletion doesn’t delete the itemset from PFP. And we update Acount and Pcount of the itemset if the itemset appears after blocks. Second is the physical deletion that Acount is less than the threshold (ε-r). The physical deletion deletes the itemset from PFP.

Mining Recent Frequent Itemsets over Data Streams

65

Fig. 1. Before and after sliding window

Definition 1. Let the minimum support be ε and the error rate be r. the itemset c is discarded from the potential frequent itemsets only if it satisfies the following formula.

L = {c | c.Pcount + c. Acount < ε × Σi , c ∈ PFP} P = {g | g. Acount < (ε − r ) × Σi , g ∈ PFP} where, i > 0 If there are a set of itemsets L that doesn’t satisfy the logical deletion, some itemsets are not frequent at this window, but frequent at the future window. So we must filter these itemsets from L. We filter the frequent itemsets from the potential frequent itemsets, we only use Acount and that is bigger than the threshold ε. Then we only delete the itemsets that whose Acount is less than the threshold (ε-r). Then we can guarantee to minimize the missing true answers and dynamically control the level of the infrequent itemsets by user.

Fig. 2. The count state in two cases

Theorem 1. Let AC be the accurate count between our proposed method and timesensitive sliding window model [12]. And j is the block point that the itemset first appears at TSi. Then the boundary of AC is as follows.

0 < AC < r × Σi −

j −1

∑ω ε

k =i −

Bk

Proof

1) Our proposed method j −1

i

If Ci ≥ ε Bi , then Acount = ∑ Ck and Pcount = k= j

∑ω

k =i −

+1

ε Bk

2) Mining and Discounting in [12] If Ci ≥ ε Bi , then Acount ' = Ci and Pcount ' =

i −1

∑ω

k =i −

+1

ε Bk

3) The accurate count AC is as following.

AC = ( Acount '+ Pcount ') − ( Acount + Pcount ) = (C i +

i −1

∑ω

k =i−

i

+1

ε Bk ) − ( ∑ C k + k= j

j −1

∑ω

k =i−

+1

ε Bk ) =

i −1

∑ε k= j

i −1

Bk − ∑ C k k= j

66

L. Jin et al.

4) From Definition 1, the boundary of Acount is as follows. ⎧ Acount + Pcount < ε × Σi −1 (1) ⎨ (2) ⎩ Acount ≥ (ε − r ) × Σi −1 From (1), Acount < ε × Σi −1 −

j −1

∑ω ε B

k =i −

k

i −1

= ∑ ε Bk k= j

i −1

∴ (ε − r ) × Σi −1 ≤ Acount < ∑ ε Bk k= j

5) From 4), the boundary of AC is as follows. i −1

i −1

i −1

i −1

k= j

k= j

k= j

0 < ∑ ε Bk − ∑ Ck ≤ ∑ ε Bk − (ε − r ) × Σi −1 = ∑ ε Bk − ε × Σ i −1 + r × Σ i −1 k= j

i −1

= r × Σi −1 + ∑ ε Bk − k= j

∴ 0 < AC ≤ r × Σi −1 −

i −1

∑ω ε

k =i −

Bk = r × Σi −1 −

j −1

∑ω ε

k =i −

Bk

j −1

∑ω ε

k =i −

Bk

The efficient discounting method has two advantages. First, the information about Acount is maintained. Acount is already computed and substituted Acount for Pcount. So the accurary of Acount is increased, e.g. Acount' at the (b) of Fig.1. Second, many missing true answers are decreased. For maintaining the information about Acount before blocks, we compute Acount more accurately and decrease the information about Acount at the blocks that doesn’t satisfy the minimum threshold. Filtering the frequent itemsets, we only use Acount and minimize many missing true frequent itemsets. The efficient discounting method has the disadvantages, too. There are many potential frequent itemsets by deleting the itemsets that satisfies the physical deletion. So there spend much memory space than [12]. For solving this problem, we propose a Sketch data structure and describe it in next section. 3.2 Sketch Data Structure

The size of sliding window must have at least 2 and have more than several basic blocks at general case. The count of any frequent itemset is at most p at each basic block. The time period p is not too big, because the input stream is divided into several basic blocks and this is the basic unit. So the size of the basic block couldn’t be very big, e.g. the number of ciphers is not bigger than 4~6 in general case. Definition 2. Let the number of basic block’s ciphers be |p|, p ≥ 1, and |ω|>1. Then the following formula is satisfied.

ω ≤ p ⇒1≤ p ≤ ω

(1)

Proof. The stream data is continuously inputted and the size of basic block couldn’t be very big. So the time period of basic block is as follows in a general case. (i) The

Mining Recent Frequent Itemsets over Data Streams

67

time period of basic block is 1 minute, then p=60 and |p|=2. (ii) The time period of basic block is 1 hour, then p=3600 and |p|=4. (iii) The time period of basic block is 1 day, then p=86400 and |p|=5. So the window size ω is more than 4~6 in general case and the formula (1) is materialized. The memory space is saved more and more according to increasing the size of sliding window ω. Definition 3. Let the frequent itemset count be C and the number of ith cipher on C is ai. Then the array of any frequent itemset count A[] is as follows: ⎧⎪ Ai + (Ct /10i −1 )%10*10 ω −t , 0 < t ≤ ω Ai = ⎨ ω −1 i −1 ⎪⎩ Ai %10 *10 + (Ct /10 )%10, t > ω where, 1 ≤ i ≤ p and 1 ≤ k ≤ ω

Proof

1) The number of ith cipher is as follows. ai = (C /10i−1 )%10, 1 ≤ i ≤ p

2) The number of end cipher is as follows at time t.

(b) case 2: t > |ω|

(a) case 1: 0 < t < |ω|

Fig. 3. The state of accumulated count

Case 1: 0 < t ≤ ω ait = (Ct /10i −1 )%10*10 ω −t , 1 ≤ i ≤ p Case 2 : t > ω aiω = (Ct /10i −1 )%10, 1 ≤ i ≤ p and 1 ≤ k ≤ ω

3) The array of any frequent itemset count A[] is as follows by from 2). Ai = Ai + (Ct /10i −1 )%10*10 Ai = Ai %10

ω −1

ω −t

, 0
*10 + (Ct /10 )%10, t > ω i −1

68

L. Jin et al.

Let |p| be 3 and |ω| be 5. Fig. 4 is the example of definition 3. The (a) of Fig. 4 is the example that the buffer isn’t full. And the (b) of Fig. 4 is the example that the sliding window starts to slide on the buffer.

(a) case 1: 0 < t < |ω|

(b) case 2: t > |ω|

Fig. 4. The example of definition 3

The detailed algorithm of definition 3 is shown as follows in Fig. 5. This structure has several advantages. (i) The size of count space only depends on |p| and isn’t on |ω|. So much space is saved than that of [12]. (ii) It doesn’t need to have the discount table (DT) as in [12]. (iii) It saves the computing time of DT.

Fig. 5. AC_List Algorithm

4 Algorithms and Examples Mining frequent itemsets in TSi consists of three steps. First, the frequent itemsets in Bi are mined by using FP-growth[18] and added into PFP with their potential counts computed. If the itemset exists in PFP, it is added into PFP and discounted from its potential count in Bi-|ω|. Second, for each itemset that is in PFP but not frequent in Bi, we scan Bi to accumulate its support count and discount the potential count of each itemset in Bi-|ω|. And we delete it from PFP if it is not frequent in TSi. Last, the frequent itemsets are outputted. The detail of algorithm is shown in Fig. 6. There are two cases: first time and other time. If first block is inserted, we only insert the frequent itemset into PFP. Other time, we not only insert the frequent itemset but also update the itemset that is not frequent in this block but already exist in PFP.

Mining Recent Frequent Itemsets over Data Streams

69

Fig. 6. Main Algorithm

4.1 Main Operations

Fig. 7 is the detail of the main operations of main algorithm. First procedure is New_item_insertion. First step is that we adopt the FP-growth algorithm to mine all the frequent itemsets from Bi. Let Fi denote this set. Next, we check each itemset in Fi to see whether it has been kept in PFP and then either update or create an entry. There are Acount and Pcount. Acount. We compute Acount by AC_List algorithm and estimate Pcount as follows.

⎡ω ⎤ Pcount at TSi = ⎡⎢ε × Σi −1 ⎤⎥ − 1 = ⎢ ∑ ε × SA[ j ]⎥ − 1 ⎢ j =1 ⎥

(2)

Second procedure is Pcount_discounting. We use this procedure only if the time point i is greater than |ω| and slide on buffer. Each itemset uses Pcount to keep its maximum possible count in the past basic blocks before it is inserted into PFP. By formula (2), since Bi comes, Pcount is computed by including the support count threshold of an extra basic block, i.e., Bi-|ω|. As Bi+1 comes, if Pcount is nonzero, we subtract the support count threshold of Bi-|ω| from Pcount. If Pcount is smaller than the support count threshold of Bi-|ω|+1, Acount should have the exact support counts from Bi-|ω|+2 to Bi-1. In this case, we set Pcount to 0. Third procedure is Old_itemset_update. For each itemset g that has been kept by PFP but not in Fi, we compute its support count in Bi to increase its Acount. If g.Acount is zero, we insert it, too. Suppose that g was inserted into PFP when Bk comes (k < i). At this point, we have g.Acount, the exact support count of g in [Bk, …, Bi], and g.Pcount, the maximum possible support count of g in [Bi-|ω|+1, …, Bk-1]. If the sum is less than the support count threshold ε, g may be frequent at the future buffer and we don’t delete g from PFP. And we continue to check that if g.Acount is less than the support count threshold (ε-r), g must not be frequent in TSi and can be safely deleted from PFP. Forth procedure is Frequent_itemset_output. It supports two kinds of outputs: NFD and NFA. NFD is the no-false-dismissal model and it guarantees that no true answer

70

L. Jin et al.

is missed. In this mode, an itemset that is frequent in Bi but not in TSi is still outputted. Sometimes user hopes that all the itemsets outputted are real answers. Therefore, we also provide the no-false-alarm mode (denoted as NFA), which outputs only the itemsets with Acount satisfying the support count threshold. Since Acount accumulates the support counts of an itemset in the individual basic blocks after that itemset is inserted into PFP, this mode guarantees that no false answer is outputted.

Fig. 7. Main Operations

4.2 Examples

Let ε, |ω|, r, and p be 0.4, 4, 0.1, and 60 minutes, respectively. Consider the stream of transactions shown in Table 1. Table 1. A stream of transactions

B1 B2 B3 B4 B5 B6

Time period

Number of transactions

Time Stamp

09:00 ~ 09:59 10:00 ~ 10:59 11:00 ~ 11:59 12:00 ~ 12:59 13:00 ~ 13:59 14:00 ~ 14:59

27 20 27 23 30 22

a(11), b(20), c(9), d(9), ab(6) a(20), c(15), d(6), ac(15) a(19), b(9), c(7), d(9), ac(7), bd(9) a(10), c(7), d(15) a(20), b(19), c(20), d(19), ac(20), bd(19) a(9), b(12), c(6), d(12), ab(3), bd(12)

Mining Recent Frequent Itemsets over Data Streams

71

Fig. 8. The snapshots after processing each basic block

Fig. 8 is the illustration of mining the frequent itemsets using our proposed method in Table 1 and shows the snapshot of the potential frequent itemsets and the support array after processing each basic block.

5 Experiment and Evaluation In this section, we perform a series of experiments for evaluating the performance of TFP-tree with the classical frequent pattern mining algorithms on synthetic data. All the experiments are performed on a Windows XP with Pentium PC 2.8GHz and 512 Mbytes of main memory. Also we use JDK 1.4, MS-SQL 2000 database and JDBC driver for connecting MS-SQL 2000. We generate three dataset, DT1, DT2, and DT3, by using our data generation program. The parameters of the number of distinct itemset, the average transaction length, the average length of the maximum pattern, and total number of transaction are respectively DT1(40, 8, 4, 10000), DT2(55, 12, 6, 10000), and DT3(45, 10, 5, 10000). For examining the performance, the basic block is 60, the minimum support is 0.6, and the error rates are 0.1, 0.2, and 0.3. The mining algorithm we applied to find frequent itemsets in a block is the FP-growth. There are two kinds of the experiments: the evaluation of our proposed method and the comparing our proposed method with others. There are two compared algorithm: TSM(Selective adjustment) and TSM-S(Self-adjustment) in [12]. Fig. 9 is the first experiment. The (a) of Fig. 9 is the experiment of accuracy at the minimum support 0.6 and the window size 4. The numbers of frequent itemsets is increasing according to increasing of the error rate, but some dataset doesn’t increase sharply like DT2. This reason is that the frequent itemsets is close to the round of the minimum support. The (b) of Fig. 9 is first experiment of saving the memory at the minimum support 0.6 and the error rate 0.1. The usage of memory is increasing slowly according to increasing of the window size. And total usage of memory is smooth. The (c) of Fig. 9 is second experiment of saving the memory at the minimum support 0.6 and the window size 4. The usage of memory is increasing smoothly

72

L. Jin et al.

according to increasing of the error rate. This proves that our proposed method doesn’t use much memory and the total memory is close to first block at some times. Fig. 10 is the experiment of comparing our proposed method with others. The (a) of Fig. 10 is the evaluation of accuracy. From the experiment, our propose method is finding the frequent itemsets more than TSM. The (b) of Fig. 10 is the evaluation of using the memory. Our propose method is saving the memory more than others. The (c) of Fig. 10 is the evaluation of the execution time. From the figure, the execution of our proposed method is smaller than others, too.

(a) The experiment of accuracy

(a) The compare of the accuracy

(b) First experiment of saving the memory

(b) The compare of saving the memory

(c) Second experiment of saving the memory

(c) The compare of saving the execution time

Fig. 9. The evaluation of our proposed method

Fig. 10. The comparing our proposed method with others

Through the above evaluation, our proposed method is more accurate and saves the memory and the execution time.

6 Conclusion We have proposed an efficient discounting method and a novel Sketch data structure for efficient mining of frequent itemsets in time-sensitive sliding-window model. And we proposed the detail of efficient algorithm. Through the experiments, our proposed method has several advantages. (i) The accuracy is increased compared with that of previous techniques. From the experiment, the efficient discounting method not only

Mining Recent Frequent Itemsets over Data Streams

73

loses the information about Acount but also decreases many missing of true answers. (ii) The memory is saved. The Sketch data structure saves much space than that of [12]. (iii) It doesn’t need to have the discount table (DT) and saves the computing time of DT. In our future work, we are going to apply the proposed method to the real application systems and evaluate our proposed method with other algorithms.

Acknowledgment This work was partially supported by the RRC program of MOCIE and ITEP, the Korea Research Foundation Grant by the Korean Government (the Regional Research Universities Program/Chungbuk BIT Research-Oriented University Consortium), and by Telematics ⋅ USN Division of ETRI in Korea.

References 1. Garofalakis, M., Gehrke, J., Rastogi, R.: Querying and mining data streams: you only get one look. In: the tutorial notes of the 28th Int’l Conf. on Very Large Data Bases (VLDB) (2002) 2. Emaine, E., Lopez-Ortiz, A., Munro, J.I.: Frequency estimation of internet packet streams with limited space. In: Proc. of European Symp. On Algorithms (ESA) (2002) 3. Karp, R.M., Papadimitriou, C.H., Shenker, S.: A simple algorithm for finding frequent elements in streams and bags, ACM Trans. On Database Systems (TODS) (2003) 4. Manku, G., Motwani, R.: Approximate frequency counts over data streams. In: Proc. of the Int’l Conf. on Very Large Data Bases (VLDB) (2002) 5. Chang, J.H., Lee, W.S.: Finding recent frequent itemsets adaptively over online data streams. In: Proc. of ACM SIGKDD Conf. (2003) 6. Cohen, E., Strauss, M.: Maintaining time decaying stream aggregates. In: Proc. of PODS Symp. (2003) 7. Giannella, C., Han, J., Pei, J., Yu, P.S.: Mining frequent patterns in data streams at multiple time granularities. In:Kargupta, H., Joshi, A., Sivakumar, K., Yesha, Y. (eds.) Next Generation Data Mining (2003) 8. Babcock, B., Datar, M., Motwani, R., O’Callaghan, L.: Maintaining variance and kMedians over data stream windows. In: Proc. of ACM PODS Symp (2003) 9. Charikar, M., Chen, K., Farach-Colton, M.: Finding frequent items in data streams. In: Proc. of ICALP (2002) 10. Cormode, G., Muthukrishnan, S.: What’s hot and what’s not: tracking most frequent items dynamically. In: Proc. of ACM PODS Symp. (2003) 11. Jin, C., Qian, W., Sha, C., Yu, J.X., Zhou, A.: Dynamically maintaining frequent items over a data stream. In: Proc. of ACM CIKM Conf. (2003) 12. Lin, C.H., Chiu, D.Y., Wu, Y.H., Chen, A.L.P.: Mining frequent itemsets from data streams with a time-sensitive sliding window, SIAM Inter’l Conf. on Data Mining (2005) 13. Han, J., Pei, J., Yin, Y., Mao, R.: Mining frequent patterns without candidate generation. In: Proc. of ACM SIGMOD Int’l Conf. Management of Data (2000) 14. Jin, L., Lee, Y., Seo, S., Ryu, K.H.: Discovery of Temporal Frequent Patterns using TFPtree. In: Yu, J.X., Kitsuregawa, M., Leong, H.V. (eds.) WAIM 2006. LNCS, vol. 4016, Springer, Heidelberg (2006) 15. Seo, S., Jin, L., Lee, J.W., Ryu, K.H.: Similar Pattern Discovery using Calendar Concept Hierarchy in Time Series Data. In: Proc. of APWeb Conf. (2004)

Mining Web Transaction Patterns in an Electronic Commerce Environment Yue-Shi Lee and Show-Jane Yen Department of Computer Science and Information Engineering, Ming Chuan University 5 The-Ming Rd., Gwei Shan District, Taoyuan County 333, Taiwan {leeys,sjyen}@mcu.edu.tw

Abstract. Association rule mining discovers most of the users’ purchasing behaviors from transaction database. Association rules are valuable for crossmarking and attached mailing applications. Other applications include catalog design, add-on sales, store layout, and customer segmentation based on buying patterns. Web traversal pattern mining discovers most of the users’ access patterns from web logs. This information can provide navigation suggestions for web users such that appropriate actions can be adopted. Web transaction pattern mining discovers not only the pure navigation behaviors but also the purchasing behaviors of customers. In this paper, we propose an algorithm IWA (Integrating Web traversal patterns and Association rules) for mining web transaction patterns in the electronic commerce environment. Our IWA algorithm takes both the traveling and purchasing behaviors of customers into consideration at the same time. The experimental results show that IWA algorithm can simultaneously and efficiently discover traveling and purchasing behaviors for most of customers.

1 Introduction As the electronic commerce (EC) activities become more and more diverse, it is very critical to provide the right information to the right customers. Web mining [4, 6, 7, 14, 15, 16, 17] refers to extracting useful information and knowledge from web logs, which applies data mining techniques [11, 12, 13] in large amount of web data to improve the web services. Mining web traversal patterns [16] is to discover most of users’ access patterns from web logs. These patterns can be used to improve website design, such as provide efficient access between highly correlated objects, and better authoring design for web pages, and provide navigation suggestions for web users. However, this information just considered the navigation behaviors of users. In EC environment, it is very important to find user purchasing behaviors. Association rule mining [1, 2, 3, 10] can discover the information about user purchasing behaviors. There are many researches worked on this field, e.g., Apriori algorithm [1], DHP (Direct Hashing and Pruning) algorithm [2], FP-Growth (Frequent PatternGrowth) algorithm [3] and H-mine (Hyper-Structure Mining) algorithm [10]. However, association rule mining just purely considers the purchasing behaviors of customers. It is important to consider navigation behaviors and purchasing behaviors of customers simultaneously in EC environment. Web transaction patterns [9] K.C. Chang et al. (Eds.): APWeb/WAIM 2007 Ws, LNCS 4537, pp. 74–85, 2007. © Springer-Verlag Berlin Heidelberg 2007

Mining Web Transaction Patterns in an Electronic Commerce Environment

75

combine web traversal patterns and association rules to provide more information for web site managers, such as putting advertisements in proper places, better customer classification, and behavior analysis, etc. In the following, we describe the definitions about web transaction patterns: Let W = {x1, x2, …, xn} be a set of all web pages in a website and I = {i1, i2, …, im} be a set of all purchased items. A traversal sequence S = <w1, w2, …, wp> (wi ∈ W, 1 ≤ i ≤ p) is a list of web pages which is ordered by traversal time, and each web page can repeatedly appear in a traversal sequence, that is, backward references are also included in a traversal sequence. For example, if there is a path which visit web page B, and then go to web page G and A sequentially, and come back to web page B, and then visit web page C. The sequence is a traversal sequence. The length of a traversal sequence S is the total number of visited web pages in S. A traversal sequence with length l is called an l-traversal sequence. For example, if there is a traversal sequence α = , the length of α is 6 and we call α a 6-traversal sequence. Suppose that there are two traversal sequences α = and β = (m ≤ n). If there exists i1 < i2 < …< im, such that bi1 = a1, bi2 = a2, …bim = am, then β contains α. For instance, if there are two traversal sequences α = and β = , then α is a sub-sequence of β and β is a super-sequence of α. A transaction sequence is a traversal sequence with purchasing behaviors. A transaction sequence is denoted as <w1{i1}, w2{i2}, … wq{iq}>, in which wk ∈ W (1 ≤ k ≤ q), ij ⊂ I (1 ≤ j ≤ q) and <w1, w2, …, wq> is a traversal sequence. If there is no purchased item in a web page, then the purchased items need not be attached with the web pages. For example, for a traversal sequence , if item i is purchased on web page , and there is no purchased items on the other web pages, then the transaction sequence can be denoted as . Suppose that there are two transaction sequences α = and β = (m ≤ n). If there exists i1 < i2 < …< im, such that bi1 = a1, bi2 = a2, …bim = am, and p1 ⊆ qi1, p2 ⊆ qi2, …, pm ⊆ qim, then β contains α. Table 1. User transaction sequence database D

TID

User Transaction Sequence

1

BECAF{1}C

2

DBAC{2}AE{3}

3

BDAE

4

BDECAF{1}C

5

BAC{2}AE{3}

6

DAC{2}

A user transaction sequence database D, as shown in Table 1, contains a set of records. Each record includes traversal identifier (TID) and user transaction sequence. A user transaction sequence is a transaction sequence, which stands for a complete

76

Y.-S. Lee and S.-J. Yen

browsing and purchasing behavior by a user. For example, the first user transaction sequence means that the user purchased item 1 on page F when he/she traversed the sequence BECAFC. The support for a traversal sequence α is the ratio of user traversal sequences which contains α to the total number of user traversal sequences in D, which is denoted as Support (α). The support count of α is the number of user traversalsequences which contain α. The support for a transaction sequence β is the ratio of user transaction sequences which contains β to the total number of user transaction sequences in D. It is denoted as Support (β). The support count of β is the number of user transaction sequences which contain β. For a traversal sequence <x1, x2, …, xl>, if there is a link from xi to xi+1 (for all i, 1 ≤ i ≤ l-1) in a web site structure, then the traversal sequence is a qualified traversal sequence. A traversal sequence α is a web traversal pattern if α is a qualified traversal sequence and Support (α) ≥ min_sup, in which the min_sup is the user specified minimum support threshold. For instance, in Table 1, if we set min_sup to 80%, then Support () = 5/6 = 83.33% ≥ min_sup = 80%, and there is a link from web page B to web page E in the web site structure shown in Figure 1. Hence, is a web traversal pattern. If the length of a web traversal pattern is l, then it can be called an l-web traversal pattern. A qualified transaction sequence is a transaction sequence in which the corresponding traversal sequence is qualified. For instance, is a qualified transaction sequence according to Figure 1. A transaction sequence α is a web transaction sequence if α is a qualified transaction sequence and Support (α) ≥ min_sup. For example, in Table 1, suppose min_sup is set to 50%, then Support () = 3/6 = 50% ≥ min_sup = 50%, and there is a link from page A to page C in the web site structure in Figure 1. Hence, is a web transaction sequence. The length of a transaction sequence is the number of web pages in the transaction sequence. A transaction sequence of length k is called a k-transaction sequence, and a web transaction sequence of length k is called a k- web transaction sequence.

Fig. 1. Web site structure

A transaction pattern can be denoted as <x1, x2, …, xi’,…, xj’,…, xn>: xi{Ii}⇒ xj{Ij}. The confidence of a pattern <x1, x2, …, xi’,…, xj’,…, xn>: xi{Ii}⇒ xj{Ij} is the ratio of support for transaction sequence<x1, x2, …, xi{Ii},…, xj{Ij},…, xn> to support for transaction sequence <x1, x2, …, xi{Ii},…, xj,…, xn>. For a web transaction sequence <x1, x2, …, xi{Ii},…, xj{Ij},…, xn>, if the confidence of transaction pattern <x1, x2, …, xi’,…, xj’,…, xn>: xi{Ii}⇒ xj{Ij} is no less than a user-specified minimum confidence

Mining Web Transaction Patterns in an Electronic Commerce Environment

77

threshold, then this transaction pattern is a web transaction pattern, which means that when the sequence <x1, x2, …, xi,…, xj,…, xn> is traversed, users purchased item Ii on page xi, item Ij on page xj was purchased too.

2 Related Work Association rule mining [1, 2, 3, 10] is to find associations among purchased items in EC environment. Path traversal pattern mining [4, 6, 7, 8, 9, 17] discover simple path traversal patterns which there is no repeated page in the pattern, that is, there is no backward reference in the pattern. These algorithms just consider the forward references in the traversal sequence database. Hence, simple path traversal patterns are not fit in real web applications. Non-simple path traversal pattern, i.e., web traversal pattern, contains not only forward references but also backward references. This information can present user navigation behaviors completely and correctly. The related researches are MFTP (Mining Frequent Traversal Patterns) algorithm [16], and FSMiner algorithm [15]. However, web site managers would like to know not only pure navigation behaviors but also purchasing behaviors of customers. Yun and Chen [9] proposed MTS algorithm, which takes both the traveling and purchasing behaviors of customers into consideration at the same time. Figure 2 shows a user transaction sequence, which the user traversed from page A to page B and purchased item 1 on page B. Next, he went to page C, and then went to page D. Thereafter, the customer went back to page C and then went to page E on which items 3 and 4 were purchased. And then, he went back to page C and bought item 2, and so on. MTS algorithm cuts each user transaction sequence into several web transaction records, that is, when a customer has backward behavior, a web transaction record is generated. MTS algorithm discovers web transaction patterns from all the web transaction records. Besides, if there is no item purchased on the last page of a web transaction record, the web transaction record does not be generated. Table 2 shows all the web transaction records generated from the user transaction sequence in Figure 2. Because there is no item bought on pages D and I, the web transaction records and cannot be generated. However, cutting user transaction sequences into web transaction records may lead to record and discover incorrect transaction sequences and lose the information about backward references. For example, from Figure 2, we can see that the customer visited page E and purchased items 3 and 4, and then went back to page C and purchased item 2 on page C. However, the first transaction record in Table 2 shows that the customer purchased item 2 on page C and then bought items 3 and 4 on page E, which is incorrect since backward information has been lost. Thus, MTS algorithm cannot generate correct web transaction patterns based on these web transaction records. Hence, we propose an algorithm IWA for mining web transaction patterns without dividing any user transaction sequences. IWA algorithm also takes both traveling and purchasing behaviors of customers into consideration at the same time, and considers not only forward references but also backward references. Besides, IWA algorithm allows noises which exist in user transaction sequences.

78

Y.-S. Lee and S.-J. Yen

Fig. 2. A user transaction sequence Table 2. Web transaction records

Path

Purchases

ABCE

B{1}, C{2}, E{4}, E{3}

ABFGH

B{1}, H{6}

ASJL

S{7}, S{8}, L{9}

3 Mining Web Transaction Patterns In this section, we describe our web transaction pattern mining algorithm IWA according to a web site structure (e.g., Figure 1). The reason for using web site structure is to avoid the unqualified web traversal sequences to be generated in our mining process. Our algorithm IWA discovers web traversal patterns and web transaction patterns from a user transaction sequence database D (e.g., Table 1). IWA algorithm first scans user transaction sequence database D once to obtain all the 1-web traversal patterns and 1-web transaction sequences. A two-dimensional matrix M for generating 2-web traversal patterns and 2-web transaction sequences is constructed. For matrix M, the rows and columns are all the 1-web traversal patterns and 1-web transaction sequences, and each entry records support count for 2-web traversal patterns and 2-web transaction sequences from row to column. From this

Mining Web Transaction Patterns in an Electronic Commerce Environment

79

matrix, we can obtain all the 2-web traversal patterns and 2-web transaction sequences. In order to generating k-web traversal patterns and k-web transaction sequences (k ≥ 3), all the (k-1)-web traversal patterns and (k-1)-web transaction sequences are joined to generate candidate k-traversal (transaction) sequences. After counting the supports for candidate k-traversal (transaction) sequences, k-web traversal patterns and k-web transaction sequences can be discovered and user transaction sequences whose length is less than (k+1) can be deleted from user transaction sequence database D. The candidate generation method for web traversal pattern is like the join method proposed in [4]. For any two distinct web traversal patterns , say <s1, …, sk-1> and , we join them together to form a k-traversal sequence only if either <s2, …, sk-1> exactly is the same with or exactly the same with <s1, …, sk-2>. For example, candidate sequence can be generated by joining the two web traversal patterns and . For a candidate l-traversal sequence α, if a qualified length (l-1) sub-sequence of α is not a web traversal pattern, then α must not be web traversal pattern and α can be pruned. Hence, we also check all of the qualified web traversal sub-sequences with length l-1 for a candidate l-traversal sequence to reduce some unnecessary candidates. The candidate generation method for web transaction sequence is similar to the join method for generating web traversal pattern, which is divided into two parts. For the first part, a (k-1)-web traversal pattern <s1, …, sk-1> and a (k-1)-web transaction sequence can be joined to form a k-web transaction sequence < s1, u1, …, uk-1{jk-1}> only if <s2, …, sk-1> is exactly the same with . A (k-1)web traversal pattern <s1, …, sk-1> and a (k-1)-web transaction sequence can be joined to form a k-web transaction sequence < u1{j1}, …, uk-1, sk-1> only if <s1, …, sk-2> is exactly the same with . In the second part, for any two distinct (k-1)-web transaction sequences <s1{i1}, …, sk-1{ik-1}> and which the set of purchased items can be empty, we join them together to form a ktransaction sequence only if either <s2{i2}, …, sk-1{ik-1}> exactly is the same with < u1{j1}, …, uk-2{jk-2}> or exactly the same with <s1{i1}, …, sk-2 {ik-2}>. For a candidate l-traversal (transaction) sequence α, if a qualified length (l-1) subsequence of α is not a web traversal pattern (or web transaction sequence), then α must not be web traversal pattern (or web transaction sequence) and α can be pruned. Hence, we also check all of the qualified web traversal (transaction) sub-sequences with length l-1 for a candidate l-traversal (transaction) sequence to reduce some unnecessary candidates. For the above example, we need to check if is a web traversal patterns, which is a qualified traversal sequence. If is not a web traversal pattern, is also not a web traversal pattern. We do not need to check , because is an unqualified traversal sequence (i.e., there is no link from web page D to web page C in Figure 1). In the following, we use an example to explain our algorithm IWA. The web site structure W and user transaction sequence database is shown in Figure 1 and Table 1, respectively. Suppose that minimum support min_sup is set to 30%, that is, minimum support count is ⎡6×30%⎤ = ⎡1.8⎤ = 2. After scanning user transaction sequence database D, 1-web traversal patterns are , , , , <E> and , and 1-web transaction sequences are ,

80

Y.-S. Lee and S.-J. Yen

<E{3}> and . The two-dimensional matrix M is shown in Table 3, in which the notation “X” represents that there is no connection in web site structure W. For example, for entry , there is no connection from web page A to web page B. From matrix M, we can easily obtain all the 2-web traversal patterns , , , , , , , , , <EC> and , and 2-web transaction sequences , , , , and . Table 3. Matrix M

A

B

C

D

E

F

C{2}

E{3}

F{1}

A

--

X

5

X

3

2

3

2

2

B

5

--

X

2

5

X

X

2

X

C

4

X

--

X

X

X

--

X

X

D

4

1

X

--

3

X

X

1

X

E

X

X

2

X

--

X

0

---

X

F

X

X

2

X

X

--

0

X

---

C{2}

2

X

--

X

X

X

---

X

X

E{3}

X

X

0

X

--

X

0

---

X

F{1}

X

X

2

X

X

X

0

X

---

After generating all the 2-web traversal patterns and 2-web transaction sequences, the candidate 3-traversal (transaction) sequences can be generated by applying the join method on 2-web traversal patterns and 2-web transaction sequences. The generated candidate 3-traversal sequences are , , , , , , , , , , , , , , , , , <ECA> and , and the generated candidate 3transaction sequences are , , , , , , , , , , , , , , , , , and . Because there is a direct link from web page D to web page E, and transaction sequence is not a 2-web transaction sequence, transaction sequence can be pruned. After the pruning, most of the candidates can be pruned. IWA scans D to count supports for all the remaining candidates and find 3-web traversal patterns and 3-web transaction sequences. The user transaction sequences whose length is less than 4 are also deleted from D. The 3-web traversal patterns are , , , , , , , , , , , , and <ECA>, and the 3-web transaction sequences are , , , , , ,

Mining Web Transaction Patterns in an Electronic Commerce Environment

81

, , and . The user transaction sequence TID 6 in D is deleted. After applying candidate generation method, the generated candidate 4-traversal sequences are , , , , , , , , , , , <ECAC>, <ECAE> and <ECAF>, and the generated candidate 4-transaction sequences are , , , <ECAE{3}>, <ECAF{1}>, , , , , , }. After scanning database D and count supports for all the candidates, the discovered 4-web traversal patterns are , , , , , <ECAC> and <ECAF>, and 4-web transaction sequences are , <ECAF{1}>, , , , and . The user transaction sequence TID 3 in D is deleted. Finally, the 5-web transaction sequences are , , <ECAF{1}C> and , and 6-web transaction sequences is . Because there is no candidate 7-web transaction sequences generated, the mining process is terminated. Suppose that the minimum confidence is 50%. The generated web transaction patterns are : C{2} ⇒ E{3}, : C{2} ⇒ E{3} and : C{2}⇒ E{3}.

4 Experimental Results We use a real dataset which the information about renting DVD movies is stored. There are 82 web pages in the web site. We collect user traversing and purchasing behaviors from 02/18/2001 to 03/24/2001 (seven days). There are 428,596 log entries in the original dataset. After preprocessing web logs, we reorganize the original log entries into 12,157 user transaction sequences. The maximum length of these user transaction sequences is 27, the minimum length is 1, and the average length is 3.9. Table 4 and Table 5 show the execution times and the number of web transacttion patterns generated by IWA on the real dataset for different minimum supports, respectively. We also generate synthetic dataset to evaluate the performance of our algorithm. Because there is no algorithm for mining web transaction patterns without cutting user transaction sequences, that is, both forward and backward traversing and purchasing behaviors are considered, we evaluate performance of our IWA algorithm by comparing with MTS algorithm [9] which cannot deal with backward references. For synthetic dataset, the number of web pages is set to 100 and the average number of out-links for each page is set to 100×30%=30. The purchasing probability for a customer in a web page is set to 30%. We generate five synthetic datasets in which the numbers of user transaction sequences are set to 200K, 400K, 600K, 800K and 1,000K, respectively. Figure 3 and Figure 4 show the execution times and number of generated web transaction patterns for MTS and IWA, respectively, when minimum support is set to 5%. From Figure 3, we can see that IWA outperforms MTS although the number of web transaction patterns generated by IWA is more than that of MTS, which is shown

82

Y.-S. Lee and S.-J. Yen Table 4. Execution times (seconds) for IWA algorithm on real dataset

Minimum Support

k-web transaction sequences (Lk)

5%

10%

15%

20%

L1

0.03

0.02

0.02

0.02

L2

1.091

1.131

1.061

1.041

L3

0.02

0.01

0.01

0.01

L4

0.03

0.02

0.01

0.00

L5

0.04

0.02

0.00

0.00

L6

0.04

0.00

0.00

0.00

Total

1.251

1.201

1.101

1.071

Table 5. The number of web transaction patterns generated by IWA algorithm on real dataset

Minimum Support

k-web transaction sequences (Lk)

5%

10%

15%

20%

L1

14

4

4

4

L2

13

6

4

3

L3

8

6

2

1

L4

5

4

0

0

L5

3

1

0

0

L6

1

0

0

0

Total

44

21

10

8

in Figure 4. This is because MTS need to take more time to cut each user transaction sequences into transaction records, such that the web transaction patterns about backward references cannot be generated. For IWA algorithm, it is not necessary to take time to cut user transaction sequences and the complete user behaviors can be retained. Hence, IWA algorithm can discover correct and complete web transaction patterns. In Figure 3, the performance gap between IWA and MTS increases as the number of user transaction sequences increases. Figure 5 shows the execution times for MTS when minimum support is 0.05% and execution times for IWA when minimum support is 5%. Owing to MTS algorithm cuts user transaction sequences into a lot of transaction records, it need to take time to do

Execution time (sec.)

Mining Web Transaction Patterns in an Electronic Commerce Environment

900 800 700 600 500 400 300 200 100 0

MTS IWA

200K

400K

600K

800K

1,000K

Number of user transaction sequences

# of web trans. patterns

Fig. 3. Execution times for MTS and IWA algorithms 900 800 700 600 500 400 300 200 100 0

MTS IWA

200K

400K

600K

800K

1,000K

Number of user transaction sequences

Fig. 4. Number of web transaction patterns generated by MTS and IWA

Execution time (sec.)

200

MTS (min_sup=0.05%)

150

IWA min_sup=5%)

100 50 0 10

15

20

25

Average length of user transaction sequences Fig. 5. Execution times for MTS and IWA algorithms

30

83

84

Y.-S. Lee and S.-J. Yen

# of web tran. patterns

1500

MTS (min_sup=0.05%)

1200

IWA min_sup=5%)

900 600 300 0 10

15

20

25

30

Average length of user transaction sequences

Fig. 6. Number of web transaction patterns generated by MTS and IWA

the preprocessing and mining we transaction patterns from a large number of transaction records. Hence, the execution time for IWA is slightly more than that of MTS although the number of web transaction patterns generated by MTS is much less than that of IWA, which is shown in Figure 6. Figure 6 shows the number of web transaction patterns generated by MTS and IWA when minimum supports are set to 0.05% and 5%, respectively. Although minimum support for MTS is much lower than minimum support for IWA, the number of web transaction patterns generated by MTS is still less than that of IWA, since MTS cuts user transaction sequences into many short transaction records such that many web transaction patterns about backward references cannot be generated.

5 Conclusions Mining association rules [1, 2, 3, 10] only discovers associations among items. Mining web traversal patterns [15, 16] just finds navigation behaviors for most of the customers. However, we cannot acquire the information about traveling and purchasing behaviors simultaneously. Therefore, this paper proposes an algorithm IWA for mining web transaction patterns which provide not only navigation behaviors but also the purchasing behaviors for most of the customers. Our algorithm retains complete user navigation and purchasing behaviors and discovers web transaction patterns including forward and backward references. The experimental results also show that our algorithm outperforms MTS algorithm which just considers forward references and incorrect web transaction patterns may be generated.

References 1. Agrawal, R., et al.: Fast Algorithm for Mining Association Rules. In: Proceedings of the International Conference on Very Large Data Bases, pp. 487-499 (1994) 2. Park, J.S., Chen, M.S., Yu, P.S.: Using a Hash-Based Method with Transaction Trimming for Mining Association Rules. IEEE Transaction on Knowledge and Data. Engineering 9(5), 813–825 (1997)

Mining Web Transaction Patterns in an Electronic Commerce Environment

85

3. Han, J., Pei, J., Yin, Y., Mao, R.: Mining Frequent Patterns without Candidate Generation: A Frequent- Pattern Tree Approach. Data. Mining and Knowledge Discovery 8(1), 53–87 (2004) 4. Chen, M.S., Park, J.S., Yu, P.S.: Efficient Data Mining for Path Traversal Patterns in a Web Environment. IEEE Transaction on Knowledge and Data. Engineering 10(2), 209– 221 (1998) 5. Yen, S.J.: An Efficient Approach for Analyzing User Behaviors in a Web-Based Training Environment. International Journal of Distance Education Technologies 1(4), 55–71 (2003) 6. Chen, M.S., Huang, X.M., Lin, I.Y.: Capturing User Access Patterns in the Web for Data Mining. In: Proceedings of the IEEE International Conference on Tools with Artificial Intelligence, pp. 345–348 (1999) 7. Pei, J., Han, J., Mortazavi-Asl, B., Zhu, H.: Mining Access Patterns Efficiently from Web Logs. In: Proceedings of the Pacific-Asia Conference on Knowledge Discovery and Data Mining, pp. 396–407 (2000) 8. Xiao, Y., Dunham, M.H.: Efficient Mining of Traversal Patterns. IEEE Transaction on Data. and Knowledge Engineering 39(2), 191–214 (2001) 9. Yun, C.H., Chen, M.S.: Using Pattern-Join and Purchase-Combination for Mining Web Transaction Patterns in an Electronic Commerce Environment. In: Proceedings of the COMPSAC, pp. 99–104 (2000) 10. Han, J., Pei, J., Lu, H., Nishio, S., Tang, S., Yang, D.: H-Mine: Hyper-Structure Mining of Frequent Patterns in Large Databases. In: Proceedings of 2001 International Conference on Data Mining (ICDM’01), San Jose, CA (November 2001) 11. Chen, S.Y., Liu, X.: Data Mining from 1994 to 2004: an Application-Orientated Review. International Journal of Business Intelligence and Data. Mining 1(1), 4–21 (2005) 12. Ngan, S.C., Lam, T., Wong, R.C.W., Fu, A.W.C.: Mining N-most interesting itemsets without support threshold by the COFI-tree. International Journal of Business Intelligence and Data. Mining 1(1), 88–106 (2005) 13. Xiao, Y., Yao, J.F., Yang, G.: Discovering Frequent Embedded Subtree Patterns from Large Databases of Unordered Labeled Trees. International Journal of Data. Warehousing and Mining 1(2), 70–92 (2005) 14. Cooley, R., Mobasher, B., Srivastava, J.: Web Mining: Information and Pattern Discovery on the World Wide Web. In: Proceedings of IEEE International Conference on Tools with Artificial Intelligence (1997) 15. EL-Sayed, M., Ruiz, C., Rundensteiner, E.A.: FS-Miner: Efficient and Incremental Mining of Frequent Sequence Patterns in Web logs, In: Proceedings of 6th ACM International Workshop on Web Information and Data Management, pp.128–135 (2004) 16. Yen, S.J.: An Efficient Approach for Analyzing User Behaviors in a Web-Based Training Environment. International Journal of Distance Education Technologies 1(4), 55–71 (2003) 17. Yen, S.J., Lee, Y.S.: An Incremental Data Mining Algorithm for Discovering Web Access Patterns. International Journal of Business Intelligence and Data Mining (2006)

Mining Purpose-Based Authorization Strategies in Database System* Jie Song, Daling Wang, Yubin Bao, Ge Yu, and Wen Qi School of Information Science and Engineering, Northeastern University Shenyang 110004, P.R. China [email protected], {dlwang,baoyubin,yuge}@mail.neu.edu.cn, [email protected]

Abstract. With the development of computer and communication technology, access control of the resources in databases has become an issue focused by both consumers and enterprises. Moreover, the new concept of purpose-based authorization strategies is widely used instead of the traditional one of rolebased strategies. The way of acquiring the optimal authorization strategies is an important problem. In this paper, an approach of mining authorization strategies based on purpose in database system is proposed. For obtaining the optimal authorization strategies of the resources in databases for supporting various purposes, an algorithm of clustering purposes is designed, which is based on the inclusion relationship among resources required by the purposes. The resultant purpose hierarchy is used for guiding the initial authorization strategies. The approach provides valuable insights into the authorization strategies of database system and delivers a validation and reinforcement of initial strategies, which is helpful to the database administration. The approach can be used not only in database system, but also in any access control system such as enterprise MIS or web service composing system. Theories and experiments show that this mining approach is more effective and efficient.

1 Introduction With the rapid development of computer network and communication techniques, the access control of system resources, especially the resources in databases, has become an issue focused by both consumers and enterprises [15, 10]. Therefore, a new concept of purpose-based authorization strategies is widely used instead of the traditional one of role-based strategies. The purpose is adapted because access control strategies are concerned with which resources is used for which purposes, rather than which users are performing which actions on which resources [5, 16]. In database systems, authorizations were commonly predefined by the system administrators [9]. However, the predefined authorizations may be not accurate because of the complication of gigantic database systems with thousands of tables. Take querying request for an example, there are numerous of tables and columns, some of which are *

This work is supported by National Natural Science Foundation of China (No. 60573090, 60673139).

K.C. Chang et al. (Eds.): APWeb/WAIM 2007 Ws, LNCS 4537, pp. 86–98, 2007. © Springer-Verlag Berlin Heidelberg 2007

Mining Purpose-Based Authorization Strategies in Database System

87

joined for individual query, and different queries may be considered as different performing ways of one purpose. In this case, for a purpose, which table should be assigned to it is illegible. According to this condition, the authorization strategies are often changed when the systems run because some purposes are authorized too deficient to be performed while others are excessively authorized. How to constitute the purpose-based authorization strategies in database systems is an important problem. In this paper, a new approach is proposed to settle the problem of the purposebased authorization strategies. Consider a database system of super market as a motivation example, where system purposes are organized according to a hierarchical structure (see Fig. 1) based on the principles of generalization and specialization. The different purposes represent the different system functions and every purpose has been authorized to some tables in the database. In this structure, the relationship among purposes and that between purposes and database resources are predefined by the database administrators when the database is ready to be on line. The purposes are invariable except that the functions of the system are extended, but the relationships among them Fig. 1. A Purpose Hierarchy Structure should be adjusted according to actual status when the database system runs, so as the relationship between purposes and database resources. These modifications are ineluctable, costly and taken place more than once, the times of such modifications should be reduced. Since any modification is based on the actual running statuses, which are logged in the access logs, a mining algorithm can be used for discovering the strategies from these logs instead of artificial analysis, and thus provides the guidance for initial authorization strategies and database privacy administration. In implementation of the algorithm, a hierarchical clustering method of data mining [7, 14], which is based on the inclusion relationship among resources required by the purposes, is used in database system for mining authorization strategies to support the initialization of them. The experiment results show that with a good mining performance, this approach can provide a nice support to purpose-based authorization strategies for database systems. The rest of this paper is organized as follows. Section 2 provides a precise definition and description of the purpose-based access control model. Section 3 sets out the algorithm of authorization strategies of mining and Section 4 describes its performance. After discussing related works in section 5, Section 6 concludes the paper with an overview of current and future research.

2 Model Definition In this section, the model of purpose-based authorization proposed in this paper is well defined using formalization definitions. In this model, the most important notions are resources and purposes. The all purposes are initialized by a set of the accessible

88

J. Song et al.

resources, and all of them are organized as a hierarchical structure, which is revised by strategies mining results. The notion of purpose plays a central role because the purpose is the basic concept on which access decisions are made. For simplifying the management and appropriate for common business environments, the purposes are organized according to a hierarchical structure based on the principles of generalization and specialization. There is one-to-multi relationship between the purpose and resources. Definition 1 (Resources). A resource indicates the data of a system and the way to access the data. In a broad sense, resources indicate tables, views in a database, or even file, operation, function, service, sub-system or something which can be requested, ignored there is a containment relationship. So a request is for one or multi resources. In this case, whatever requests, each of them is regarded as requesting a resource set. Let set R present the set of all resources in the system. R = {ri | 0 < i < n} Where n means the number of resources in the system. Definition 2 (Purpose and Purpose Tree). A purpose describes the intention(s) for accessing some resources. A purpose set is organized as a hierarchical structure, referred to as Purpose Tree, where each node represents a purpose in the purpose set and each edge represents a hierarchical relationship (specialization and generalization) between two purposes. Let PTree be a Purpose Tree and P be the purpose set in PTree. (Where n means the number of purposes).

PTree = P I {relationship between pi and p j | pi , p j ∈ P} (0 < i , j < n) In the purpose-based authorization model, an accessible resource set is assigned to one purpose for supporting the positive privacy strategies of the purpose, which means the resources that the purpose can access are included in this set. This mechanism is similar to the famous RBAC model [8, 12]. The relative definitions are given as follows. Definition 3 (Accessible Resources). Accessible Resources (AR) defines the resources one purpose can access. By using Accessible Resources, one can guarantee that accesses to particular resources are allowed and on the contrary the others are rejected according to a certain purpose. Two AR sets may consist of the same resources, and thus provide greater flexibility. Evidently, AR ⊆ R: (0 < i < n ) ⎧ {r | r ∈ R } AR = ⎨ i i ( no resource is permited) ⎩∅

Where n is the number of accessible resource. In purpose-based authorization model, one purpose corresponds with one AR. Intuitively, based on the hierarchical structure of purposes, ARp of one purpose includes not only the AR set assigned to itself, but also the AR sets assigned to its ancestors. If ARp is an Accessible Resources that purpose p is authorized, then: ARp = ( U pi∈P AR of Ancestor(pi)) (p ∈ Ancestor(p) ) The strategies of verification are based on what kind of AR the purpose has, so the verification strategies are straightforward. ∀ p ∈ P, r ∈ R, ARp is Accessible Resources of p, the condition that purpose p can access resource r is r ∈ AR.

Mining Purpose-Based Authorization Strategies in Database System

89

3 Authorization Strategies Mining As discussed in the previous section, a purpose-based authorization model needs to be built up. In this section, the main idea of this paper, authorization mining is formally described. There are two steps of this approach: analyzing permission and mining purpose hierarchy. And the result of the strategies should be well used as a guidance of initial strategies. z Permission Analyzing Any person acting with a purpose has to access some resources in order to perform it, and there are many ways to fulfill the same purpose. However, the best way is the one that needs the least resources according to information hiding principle, and these resources are actual AR of the purpose contrast to initial AR. In this section, the approach of extracting the best AR set by analyzing access logs is described. Definition 4 (Logs and Record). The purposes performing statuses are recorded in logs, which contain many records. Each record has three parts: purpose, requested resources and their access statuses. Logs = { record i }

record = < p j ,{< rk , status > } >

( p j ∈ P, rk ∈ R, 0 < i , j , k < n, staus ∈ {true, false})

Here rk is one of the resources used to perform the purpose, and a status has the alternative values: true or false, which means pj can or cannot access rk. For example, a user chooses several resources to accomplish a purpose, and then these resources are verified. If the purpose does not have permission to access the resource, a tuple is recorded in logs. Since the way of a user to perform one purpose is different, the resource he chooses is also different. Definition 5 (Record Frequency). Record Frequency (RF) is the frequency of a specified record occurs within the records, which belong to the same purpose. If p is a purpose and p ∈ P, re is one of the records in Logs, re∈Logs:

RF p =

| {record i | record i = re} | | {record i | purpose of record i is p} |

Definition 6 (Permission Set). Permission Set is extracted from Logs as the result of permission analyzing, which is the best mapping relationship between purposes and resources, and is used in next purpose hierarchy mining. Let PS be the Permission Set, then psi is a two-tuple , PS satisfies: PS = { psi } = {< pi , AR p i >} AR p i is resurce set of reasonable record of pi

| PS | = | P |

There are several steps to generate permission set. First, rational records that can fulfill a certain purpose are extracted from logs. Of these records, those happened seldom or RFs less than a certain value are omitted. Table 1 shows the relationship between records and resources for a certain purpose as example. In the table, “√” means the resource has been assigned to the purpose, and “×” means the resource

90

J. Song et al. Table 1. All Records of A Certain Purpose record1 record2 record3 record4

r1 √ √ √ √

r2 √ √ √ √

r3 × ×

r4

r5

r6

r7

√

√

√

× ×

RF rf1 rf2 rf3 rf4

needs to be performed for the purpose but has not been authorized. The values in the last row of the table show the RFs of each record. Then, the analyzing algorithm is following several rules to extract the permission set of each purpose: Rule 1: According to the information hiding principle, find the record which needs the least resources. If several records are selected, then follow the rule 2. Rule 2: Because of minimizing resource allocation, resource retraction is advocated while re-assignment should try to be avoided, so the record which can fulfill the purpose is selected first. That is, all the resources it needs are marked “√”. If there is no purpose satisfied this condition, then follow the rule 3, or more purposes satisfied, follow the rule 4. Rule 3: If resource re-assignment is ineluctable, select the record with least “×” in order to furthest restrict the authorization. If an exclusive record cannot be selected by this rule, then follow rule 4. Rule 4: The higher the RF of record is, the more often the record is chosen to perform the purpose. For adapting the habit of most users, the record with maximal RF is selected. If there are several records with equal RFs, then select one randomly.

In table 1, record1 and record2 are selected by rule 1. Since rule 2 and rule 3 cannot find the result, an exclusive record1 is select at last by rule 4 (RF1>RF2). The detail of permission analyzing algorithm is shown as algorithm 1. Algorithm 1. Permission Analyzing Algorithm Input: Logs and P Output: PS Analyzing(): 1. PS = ∅ 2. For each p of P 3. recordp = all records of p in Logs; 4. Delete the records from recordp which RF of them < min_value; 5. Delete the records from recordp which resources of them > max_value; 6. If one or more records in recordp can fulfill the purpose 7. Find the record with the largest RF; 8. Add record to PS; 9. Break; 10. End If 11. Find records that have the least “×”; 12. Find the record with the largest RF; 13. Add record to PS; 14. End For 15. Return PS ;

Mining Purpose-Based Authorization Strategies in Database System

91

z Purpose Hierarchy Mining When permission set of each purpose is extracted from a number of logs by the algorithm proposed in previous section, this PS is prepared for the next purpose hierarchy mining. In this section, a purposes hierarchical clustering method based on inclusion relationship among the resources required by the purposes is proposed for mining the purpose hierarchy. In detail, the idea of the algorithm is that the less resource a purpose requires, the higher level the purpose is in the hierarchy. The resources required by the same level purposes have not inclusion relationship. According to the idea, some definitions related with the clustering and the algorithms are given here. Definition 7 (Overlap). ∀pi , p j ∈ P (i ≠ j ) , then pi overlap pj iff ARp j ⊂ ARpi . The

overlap is signed as “≥”, so pi overlap pj means pi ≥ pj. Definition 8 (Purpose Cluster Set). The Purpose Cluster Set is the set of some purpose clusters. The purposes in one cluster have the relationship “≥”. The set is signed as PC and its every purpose cluster is signed as pici (0
and

PC = { pi ci | 0 < i < n}

Definition 9 (PY and PN). Let PY be the set of purposes which have been clustered, and PN be the set of purposes which haven’t been clustered. Then:

PY ⊆ P and PN ⊆ P and PY I PN = ∅ and PY U PN = P Obviously, because of pi ≥ pj, there is such purpose cluster as (pi,(pjcj)). The “⊂” between resources required by some purposes results in the “≥” between the purposes, and the “≥” generates the hierarchy of the purposes. According to the relationship, the purpose hierarchy can be built by mining purpose clusters based on the “⊂” between resources required by the purposes in every cluster. Followings are the steps of the purpose hierarchy mining algorithm. Step 1. Go through all PS to see if there are several purposes that be assigned the same resources. If there are, combine the purposes as one purpose; Step 2. Initialize the variables: PY=∅, PN=P, PC=∅; Step 3. From all purposes in PN, find one that doesn’t overlap any purpose in PY while requires the least resources, and put it into PY. Delete that purpose from PN and add a new corresponding cluster to the set of all clusters. After this step:

PY = PY U { p i | p i ∈ PN and p i doesn' t overlap the elements in PN while | AR p i | is minimal } PN = PN I { p i } PC = PC U { p i c i | 0 < i < n}

92

J. Song et al.

Step 4. Repeat step 3 until there is no purpose in PN that doesn’t overlap any purpose in PY; Step 5. Now, each purpose in PN should overlap at least one of the purposes in PY. Find one purpose (pi for example) in PY that requires the least resources, then traverse all purposes in each cluster from lower level to higher level until finding the first one (pj for example) which is overlapped by pi. If pj is at the lowest level, add a new lower level to the corresponding cluster (pcj) in PC. And if pj isn’t at the lowest level, then add pi to pj‘s sub-level. Add pi to PY and remove it from PN. After this step:

PY = PY U { pi | pi ∈ PN and | AR p i | is minimal} PN = PN I { pi }

p j c j = ( p j ( pi ci )) Step 6. Repeat step 5 until there is no element in PN. Step 7. Now the PTree can be constructed according to Purpose Cluster Set. First, add a general purpose to the PTree as a root node, and then every pc in PC is a subtree of it. For each pc, while pc is not empty and there is a level, add the first purpose of this level to its super-node as a sub-node and the purposes of the same level are its brother-node. When all pc in Purpose Cluster Set have been added to the root node, the PTree is constructed.

The detail of purpose hierarchy mining algorithm is shown as algorithm 2 and algorithm 3. Algorithm 2. Mining Cluster Algorithm Input: P Output: PTree Mining(): 1. Sort P set by AR of each p; 2. Combine purposes whose AR is the same as one purpose; 3. PTree = ∅ , PY= ∅ , PN= P, PC = ∅ 4. For each p of PN from begin to end // PN or P 5. //have be sorted, so AR of pi in PN is minimal 6. If AR of pi in temp is minimal, 7. Add pi to PY and delete pi from PN and add pici to PC; 8. End if 9. Else 10. For each cj in PC 11. While picj is not empty 12. templevel = the lowest level of picj; 13. If pj is at templevel and pj is overlapped by pi 14. Add pi to pj’s lower level; 15. Add pi to PY, delete pi from PN; 16. End If 17. templevel = templevel’s super-level; 18. End While 19 End For 20 End Else 21. Return buildingTree(PC);

Mining Purpose-Based Authorization Strategies in Database System

93

Algorithm 3. Building Purpose Hierarchy Algorithm Input: PC Output: PTree buildingTree(): 1. Add a general purpose to PTree as root node; 2. For each pc in PC 3. While pc ≠ ∅ 4. If p is each level’s first purpose 5. add a sub-node to its super-node of PTree 6. End If 7. Else 8. add a brother-node to the same level’s purposes; 9. End Else 10. End While 11. End For 12. Return PTree;

Example (Supermarket Database). In this example, the purpose tree of the motivation example is shown as the left part of Fig. 2, in which the true name of each purpose is p0 p0 replaced by pi in order to present tersely. And table 2 p1 p2 p3 p3 p2 p1 {r2,r3…} {r2,r4…} {r4…} {r2,r3…} shows the best record of each purpose as the result p7 p8 p9 p7 p9&6 p8 p4 p5 analyzed by algorithm 1. p4 p5 p6 {r2,r3,r4,r9...} {r2,r4,r9...} {r2,r4,r9...} Now apply the algorithm 2 p p p p 10 10 11 11 to construct the cluster hierarchy as following Fig. 2. Purpose Hierarchy of the Example items: Table 2. Analyzed Result of Motivation Example p1 p2 p3 p4 p5 p6 p7 p8 p9 p10 p11

r1 √

r2

r3

√

√

√ √ √

√ √

r4

r5

r6

r7

r8

r9

r10

r11

√

× √ ×

× √ × √ √ √ √ √

√ √ √ √ √ √

√ √

1. In the first two steps, variables are initialized: PN={p1,p2,p3,p4,p5,p6&9,p7,p8,p9,p10,p11} Here, since p6 and p9 require the same resources, they are combined as p6&9; 2. Step 3 adds the first cluster(pc) to the PC, and because p1 and p3 only require one recourse, they are the first elements that be added to PY. And the rest are remained in PN:

94

J. Song et al.

PY = {p1,p3} PN = {p2,p4,p5,p6&9,p7,p8, p9,p10,p11} PC = {(p1),(p3)}; 3. After the iteration of step 4: PY = {p1,p3,p2} PN = {p4,p5,p6&9,p7,p8,p9, p10,p11} PC = {(p1),(p3),(p2)}; 4. In step 5, p7 requires two resources and is the one that requires the least. And p3 in PY is overlapped by it, so p7 is added to PC as p3’s sub-level’s purpose. Then add p7 to PY and delete it from PN: PY = {p1,p3,p7,p2} PN = {p4,p5,p6&9,p8,p9,p10,p11} PC = {(p1),(p3 (,p7)),(p2)}; 5. The iteration of step 6, the result is: PY = {p1,p3,p7,p10,p11,p6&9,p8,p2,p4,p5} PN = ∅ PC = {(p1),(p3 (,p7(,p10,p11) ,(p6&9),(p8)),(p2(p4,p5))}; Since PN has no element, no cluster is created. So, PC is stable and algorithm stops. Now the cluster hierarchy is got and a purpose hierarchy is derived from it. The right part of Fig. 2 shows the PTree of the example. z Result Explaining The purpose hierarchy shows that purpose can access to all the resources of its ancestor. It means that a specialized purpose requires more resources to be performed than a generalized one. After a careful comparing between the result tree and initial tree shown as Fig. 2, some apparent changes are made evidently. For example, the order of the sub-tree of p0 (General Purpose) is changed, while these changes do nothing with the authorization strategies. However, the other changes imply the real strategies. Some purposes are excessively authorized, such as p3 (Sales), while some purposes which can be fulfilled by the less resources have to make a detour, because they are authorized irrationally, and p6 (Returns) is an instance. Purposes those require the same resources are regarded as different because they are authorized different resources which are not appropriate, p6 (Returns) and p9 (Refundment) are of such cases. All of these against the information hiding principle cause a waste of resources and the increase of management cost. With the purpose hierarchy, the resources which a purpose actually requires are obvious. Then it can be used in database system’s authorization strategies, and make the strategies more reasonable and effective.

4 Experiments and Performance Now, the purpose-based authorization strategies mining approach is well used in National Ocean Environment Database System, which has more than 1000 tables (about ten million records each table) and 1000 purposes. All the visited SQL are use a key word “for” to indicate its purpose, and be executed by a special SQL syntax

Mining Purpose-Based Authorization Strategies in Database System

95

analyzer. Take a querying for example (updating and deleting are similar with querying), the SQL syntax is like following: SELECT [,] FROM [,] [WHERE ] FOR

8

4000

7

3500

6

3000

5 4

2500

1500

2

1000

1

500 10

20

50

70

100

Records(thousand)

Fig. 3. Results of Storage Evaluation

1000

800

600

2000

3

0

1200

50 purposes 100 purposes 500 purposes 1000 purposes 2000 purposes

Time (s)

Time (ms)

Storage(MB)

Authorization strategies mining can be done based on the log table which is chosen as a container of logs, some practical results about this database are given to prove that strategy mining is effective and efficient. The storage overhead of proposed approach is also inconspicuous. Some storage is spent on archiving logs. But the experiment in Fig.3 shows that it only costs about 1MB storage per ten thousand records. Comparing with the tremendous database with the storage level of GBs or TBs, it may as little as the storage cost of an index of small table, it can be ignored. And as for strategies mining, better result is more important than mining performance. But the performance is also gratifying as shown in Fig. 4 and Fig. 5 under variety number of records (see definition 4) and purpose involved. The results of tests prove that the performance of analyzing and mining algorithm is very good (the mining performance shown in Fig. 4 doesn’t include I/O cost for result output).

400

200

0

0 2000 5000 10000 20000 50000

Number of Records

Fig. 4. Analyzing Performance

50

100 500 1000 2000 5000

Numner of Purposes

Fig. 5. Mining Performance

Maybe the most important thing concerned by the users is the effect of strategy mining. Table 3 shows some qualitative comparisons between the way with and without authorization strategies mining. The experiment results show that this approach can reduce the times, number of purposes involved, cost and side-effect of strategies modification, so can meet the requirements of users and implement a good degree of information hiding. Moreover, with the increasing of purposes, resources, and query requests of users in database, the advantages of purpose-based authorization will more obvious.

96

J. Song et al. Table 3. Comparisons Between the Way with or without Mining Aspect

With Strategies mining

Without Strategies mining

Times of strategies modification to achieve reasonable one

2 or 3 times

More than 20 times

The average number of purposes involved in once strategies modification

About 10 purposes

1 or 2 purposes

The user satisfaction after once strategies modification

Very good

Part of them are satisfied

Time cost of one strategies modification

More times is needed

Quickly

Side-effect of one strategies modification

No side-effect

Uncertain side-effects

Degree of information hiding

Very good

Overage resources may be authorized

5 Related Work There are some researches about the authorization strategies in database system though they are not based on data mining technique. Some of them are helpful to this research. Bertino and Samarati proposed two extensions to the authorization model for relational databases, and gave the conversion from an authorizations status to another one [3]. Bertino and Jajodia presented an authorization model that can be used to express a number of discretionary access control policies for relational data management systems [4]. Afinidad and Levin proposed a model for representing temporal access control policies. In this model, temporal authorizations are represented by time attributes associated with subjects and objects [1]. Keahey and Welch, et al. proposed a fine-grain authorization for resource management[11,17]. Take Elisa Bertino [5] and Ji-Won Byun [6] built a purpose-based access control model for not only database system, but also other privacy system with complex data. In this paper, the purpose-based access control model is based on [5] and [6]. However, the issue of this paper is how to support and consummate the purpose model with data mining technology different from [5] and [6]. Data mining techniques has been widely used in database systems [2]. Since traditional top-down approach to role engineering does not analyze the permission enter in the role definition process, J¨urgen Schlegelmilch proposed RCA to solve this problem [15]. Whereas, role mining focuses mostly on the rights that a performance needs, rather than the resources a performance needs. What’s more, different role may have the same purpose, while role mining cannot figure out this deep-seated relation. Purpose mining delves into the relationship between resources and purposes, and optimizes the authorization strategies in database systems. A hierarchy mining algorithm has been proposed to commendably find out that relationship, and a purpose tree is constructed to motivate the authorization strategies. Other mining applications also support the strategies mining algorithm. Qiang Yang presented an approach of case mining to automatically discover case bases from large datasets in order to improve both the speed and the quality of case based

Mining Purpose-Based Authorization Strategies in Database System

97

reasoning [18]. Case mining constructed a case base from a large raw dataset with an objective to improve the case-base reasoning systems’ efficiency and quality. Jie Zhang presented an approach that exploits the data dimension by mining user query log to glorify the ontology learning process [19]. Since these works both propose some kinds of mining algorithm, but the strategies mining algorithm is brand-new and dose not strictly match any of them.

6 Conclusions and Future Work In this paper, an approach of purpose-based authorization strategies mining is presented and implemented, which is primarily used in database access control. Generally, compared with previous studies, purpose-based authorization strategies are more flexible and efficient. And mining algorithm operates on a more microcosmic level, reduces the waste of resources and saves the management cost, when database systems are getting larger and larger. Future work of this research includes a weighted purposes hierarchy mining algorithm, which provides a more flexible model of authorization strategies. And the ignored containment relation between resources should be well considered. And the implementation of the mining algorithm to the other privacy system such as web service, composing system, and work flow system, which need to be well considered to enhance the usage of purpose-based authorization strategies mining algorithm.

References [1] Afinidad, F., Levin, T., Irvine, C., Nguyen, T.: A Model for Temporal Interval Authorizations. HICSS (2006) [2] Apté, C., Liu, B., Pednault, E.P.D., Smyth, P.: Business applications of data mining. Commun. ACM 45(8), 49–53 (2002) [3] Bertino, E., Samarati, P., Jajodia, S.: An Extended Authorization Model for Relational Databases. IEEE Trans. Knowl. Data Eng. 9(1), 85–101 (1997) [4] Bertino, E., Jajodia, S., Samarati, P.: A Flexible Authorization Mechanism for Relational Data Management Systems. ACM Trans. Inf. Syst. 17(2), 101–140 (1999) [5] Bertino, E.: Purpose Based Access Control for Privacy Protection in Database Systems. In: Zhou, L.-z., Ooi, B.-C., Meng, X. (eds.) DASFAA 2005. LNCS, vol. 3453, 2, Springer, Heidelberg (2005) [6] Byun, J., Bertino, E., Li, N.: Purpose based access control of complex data for privacy protection. SACMAT 2005, pp. 102–110 (2005) [7] Du, Z., Lin, F.: A novel parallelization approach for hierarchical clustering. Parallel Computing 31(5), 523–527 (2005) [8] Ferraiolo, D., Sandhu, R., Gavrila, S., Kuhn, D., Chandramouli, R.: Proposed NIST Standard for Role-Based Access Control. ACM Transactions on Information and ystemsSecurity, vol. 4(3) (August 2001) [9] Griffiths, P., Wade, B.: An Authorization Mechanism for a Relational Database System. ACM TODS 1(3), 242–255 (September 1976) [10] Hitchens, M., Varadarajan, V.: Tower: A Language for Role-Based Access Control. In: Proceedings of the Strategies Workshop, Bristol, UK (2001)

98

J. Song et al.

[11] Keahey, K., Welch, V.: Fine-Grain Authorization for Resource Management in the Grid Environment. In: Parashar, M. (ed.) GRID 2002. LNCS, vol. 2536, pp. 199–206. Springer, Heidelberg (2002) [12] Kim, D., Ray, I., France, R., Li, N.: Modeling Role-Based Access Control Using Parameterized UML Models. In: Wermelinger, M., Margaria-Steffen, T. (eds.) FASE 2004. LNCS, vol. 2984, pp. 180–193. Springer, Heidelberg (2004) [13] Kumar, A., Karnik, N., Chafle, G.: Context sensitivity in role-based access control. In: ACM SIGOPS Operating Systems Review (July 2002) [14] Lee, J., Yeung, D., Tsang, E.: Hierarchical clustering based on ordinal consistency. Pattern Recognition 38(11), 1913–1925 (2005) [15] Schlegelmilch, J.: Role mining with ORCA. SACMAT 2005, pp. 168–176 (2005) [16] Tachikawa, T., Higaki, H., Takizawa, M.: Purpose-Oriented Access Control Model in Object-Based Systems. In: Mu, Y., Pieprzyk, J.P., Varadharajan, V. (eds.) ACISP 1997. LNCS, vol. 1270, pp. 38–49. Springer, Heidelberg (1997) [17] Thompson, M., Essiari, A., Keahey, K., Welch, V., Lang, S., Liu, B.: Fine-Grained Authorization for Job and Resource Management Using Akenti and the Globus Toolkit. CoRR cs.DC/0306070 (2003) [18] Yang, Q., Cheng, H.: Case Mining from Large Databases. In: Ashley, K.D., Bridge, D.G. (eds.) ICCBR 2003. LNCS, vol. 2689, pp. 691–702. Springer, Heidelberg (2003) [19] Zhang, J., Xiong, M., Yu, Y.: Mining Query Log to Assist Ontology Learning from Relational Database. In: Zhou, X., Li, J., Shen, H.T., Kitsuregawa, M., Zhang, Y. (eds.) APWeb 2006. LNCS, vol. 3841, pp. 437–448. Springer, Heidelberg (2006)

RSP-DS: Real Time Sequential Pattern Analysis over Data Streams* Ho-Seok Kim1, Jae-Jyn Shin1, Yong-Il Jang1, Gyoung-Bae Kim2, and Hae-Young Bae1 1

Department of Computer Science and Information Engineering, Inha University Yonghyun-dong, Nam-gu, Incheon, 402-751, Korea {hskim, jaejyn, yijang}@dblab.inha.ac.kr, [email protected] 2 Department of Computer Education, Seowon University, 231 Mochung-dong Heung-duk-gu Cheongju-si Chungbuk, 361-742, Korea [email protected]

Abstract. The existing pattern analysis algorithms in data streams environment have only focused on studying performance improvement and effective memory usage. But when new data streams come, existing pattern analysis algorithms have to analyze patterns again and have to regenerate pattern tree. This approach needs many calculations in real time environments having real time pattern analysis needs. This paper proposes a method that continuously analyzes patterns of incoming data streams in real time. The proposed method analyzes patterns first, and then after obtains real time patterns by updating previously analyzed patterns. The patterns form a pattern tree, and freshly created new patterns update the pattern tree. In this way, real time patterns are always maintained in the pattern tree and old patterns in the tree are deleted easily using FIFO method. The advantage of our algorithm is proved by performance comparison with existing methods, MILE, with a condition that pattern is changed continuously.

1 Introduction Many applications worthy of being noticed, including network packet monitoring, measurement of scientific experimental performance, financial management and other applications in ubiquitous computing, requiring real-time processing deal with new data type called data stream. In traditional databases the data does not change so much in time. On the other hand, in data streams data is inputted continuously with differing input volume and speed different from the previous data at every instant of time. It also may be generated from several sources at the same time[3]. The existing data mining techniques should be redefined depending on data stream’s specific characteristics such as input volume and frequency. It is impossible to store all data from the stream as enormous data is generated continuously. Therefore data mining operations *

This research was supported by the MIC (Ministry of Information and Communication), Korea, under the ITRC (Information Technology Research Center) support program supervised by the IITA (Institute of Information Technology Assessment).

K.C. Chang et al. (Eds.): APWeb/WAIM 2007 Ws, LNCS 4537, pp. 99–110, 2007. © Springer-Verlag Berlin Heidelberg 2007

100

H.-S. Kim et al.

should not use all of the data but the recent data. Also since the characteristic of data stream changes continuously, the result of mining operation should be re-analyzed. But, these absolute conditions require much memory and calculation time to analyze the pattern of large volume of data stream that is inputted with fast speed once more[10]. Sequential pattern analysis method finds the set of frequent sub-sequences is a field of data mining[11]. GSP(Generalized Sequential Patterns) and PrefixSpan are two typical methods for sequential pattern analysis[1,4]. GSP discovers sequential pattern based on the part of the pattern that happens frequently if some patterns happen frequently. But, GSP requires so much time to calculate how many times sub-sequential pattern occur, it can't be adapted to real time data streams. PrefixSpan doesn't generate the candidate patterns and discovers sequential pattern faster than GSP, so it can be adapted to data stream analysis. StreamPath and DSM-TKP(Data Stream Mining for Top-k Path Traversal Patterns) were proposed for sequential pattern analysis of data stream[6,9]. But, they have a weak point that these methods should search common short-term pattern in long-term patterns repeatedly. MILE(MIning from muLtiple strEams) is the algorithm to analyze sequential pattern in multiple streams, based on PrefixSpan[2]. MILE recursively utilizes the knowledge of existing patterns to avoid redundant data scanning, and can therefore effectively speed up the new patterns’ discovery process. As mentioned above, because of characteristic of data streams, method of sequential pattern analysis over data streams should discover pattern which change continuously. In this paper, we propose the algorithm that analyzes and discovers pattern over data streams which changes continuously. The proposed algorithm divides data streams into several sequences and makes several sequences. These sequences are managed by hash table that have data as key. In case that the regular number of sequences is entered, pattern analysis is processed in hash table in each time. Patterns that are analyzed continuously are set to pattern-tree and old pattern in pattern-tree is deleted to keep present pattern. The rest of this paper is organized as following. Section 2 reviews related work: PrefixSpan and MILE. The proposed algorithm and performance evaluation is described in Section 3. A conclusion is presented in the last section.

2 Related Work 2.1 PrefixSpan PrefixSpan is a method that discovers sequential pattern. The merits of PrefixSpan are that it recursively projects the original dataset into smaller and smaller subsets, from which patterns can be progressively mined out. PrefixSpan does not need to generate candidate patterns and identify their occurrences but grows patterns as long as the current item is frequent in the projected dataset as depicted figure 1. Therefore this algorithm is not proper for data stream environment that require fast processing time. Also, it wastes memory to store redundant patterns in pattern tree.

RSP-DS: Real Time Sequential Pattern Analysis over Data Streams

101

For example, given initial value of processing pattern and final pattern set is empty and min_sup is equal to 2 and 3 sequences are , , , First, it finds the set of frequent items over min_sup, frequent items are listed as following: a=3, b=3, c=3, d=3, e=1, f=2, g=2, h=1. To make new processing pattern, each data that is selected is merged with processing pattern. But, because now frequent data is empty, each data can be processing pattern , , , , , . Processing pattern is added to final pattern set. In case of , Suffix of processing pattern in each sequence, it is , , . Patterns as a result of recursive call functions are depicted as pattern tree as below.

Fig. 1. Pattern Tree by PrefixSpan

2.2 MILE (MIning from muLtiple strEams) In PrefixSpan, it is a demerit that a pattern already found will be generated once more. For example, in figure 1, left most pattern and its child patterns , , are listed Suffix of , and . As the size of patterns increases more, some more redundant pattern search increase, so the cost of pattern search is an expense. MILE divides a large number of data stream source into tuple based windows and forms several sequences. Unlike PrefixSpan, in MILE given pattern is linked with previous pattern if a current Suffix of pattern is already listed in the pattern tree. It prevents redundant pattern search, improves the speed of patter generation. Given sequence seq1=, seq2=, seq3= in figure 1, it forms pattern by PrefixSpan from data that is inputted later relatively to other patterns to prevent redundant pattern search. When pattern is formed, , are already listed including as Prefix. So, there is no pattern generation any more and , including as Prefix are pointed in . After forming entire patterns, pattern tree is generated like in figure 2. There are fewer patterns in pattern tree than figure 1. Pattern and have a pointer of patterns to begin by that are already found. MILE based on PrefixSpan adapts simple sequential pattern analysis to pattern analysis of data stream. But, since pattern of each data stream changes continuously, pattern frequency of each data stream may be different. MILE divides data stream into tuple based window and analyzes it.

102

H.-S. Kim et al.

Fig. 2. Pattern Tree by MILE

3 RSP-DS (Real Time Sequential Patterns Analysis in Data Streams) In RSP-DS, patterns of several sequences Si, Si+1, ..., Si+n divided by time based sliding window against previous patterns Si-1, Si-2, ... ,Si-(n-1) is analyzed. Terms used in pattern analysis is defined as followings. Definition 1: old_seqs It means the count of sequences inputted previously to discover pattern. Sequences can be inputted frequently or rarely. If patterns are discovered in sequences that are inputted continuously, we can discover patterns using only new sequences. But, in case that some patterns are discovered rarely, we have to compare with old sequences to discover those patters. Definition 2: new_seqs It means the count of new sequences to be used in pattern analysis. As comparing previous sequence to new sequence when analyzing pattern, it is new_seqs that decides how many new sequences are required. The number of (old_seqs + new_seqs) sequences is necessary for pattern analysis. And, after analyzing the pattern, old_seqs is set to old_seqs exclusive of the oldest number of new_seqs sequences. 3.1 Pattern Analysis and Addition RSP-DS uses MILE to discover patterns over a data stream. MILE is an algorithm that discovers patterns over several data stream, on the other hand, RSP-DS divides a data stream into a fixed sequence and discovers pattern over those sequences. Data Structure It is assumed that a pattern tree is made from 3 windows (, , ). When sequence is inputted, we should analyze pattern again including a newly inputted sequence exclusive of that is inputted for the first time. That is, it is assumed that old_seqs be 2 and new_seqs be 1. Figure 3 depicts pattern analysis when old_seqs=2, new_seqs=1. At time t1, , , are used for pattern analysis. But, at time t2, the rest of the sequences including the newly inputted are done exclusive of that is inputted for the first time.

RSP-DS: Real Time Sequential Pattern Analysis over Data Streams

103

Fig. 3. The change of sequences using pattern analysis according to time

When discovering new patterns, much time is demanded to compare previous pattern tree and pattern tree that is made by current sequences. Therefore, we should make new pattern tree by updating previous pattern tree partially. If we know that the Suffix of current pattern is redundant compared to previous pattern, it can prevent us from extracting redundant pattern to link pointer to previous pattern. For example, lets assume there are pattern and pattern correctly existing. If Pattern is found earlier than pattern , when pattern is found later than , simply the pointer can be linked from to . At this time, we should know which pattern is entered as Suffix of another pattern to maximize it by removing redundant trace pattern. Sequences used for analyzing pattern are managed by hash table to analyze pattern quickly. In figure 4, old_seqs is 2 and new_seqs is 1 in hash table. Sequence 3 is the last entered sequence. After sequences including seq1=, seq2= , seq3= is analyzed, hash index is constructed. Data Information Table(DIT) pointed to by hash index stores statistics information about entire sequences, and Order Information Table(OIT) stores information data of each sequence.

Fig. 4. The Example of Hash Table

Once new sequence is entered in OIT, the row in OIT corresponding to oldest sequence is deleted and a new row is inserted. OIT is composed of nOrder and pNext. The nOrder points to the order of data in a sequence and the pNext is the pointer that points to Suffix when this data in a sequence is the Prefix. It is to look up nOrder corresponding to each sequence and which is the way to decide whether Sequence is pattern or not. Because Seq1, Seq2, Seq3 sequences appear later

104

H.-S. Kim et al.

than a and b, these sequences are turned out pattern. The pNext is used for determining whether a data is linked to any other pattern when analyzing pattern. Therefore it can prevent redundant pattern analysis easily. DIT is composed of bFreq and nSuffix. The bFreq denotes the frequency of data appearing in sequences. The bFreq is used for determining whether a data appears frequently or not. The redundancy of pattern analysis can be minimized by first searching short data that is quite probable to be entered as Suffix of other pattern, by using nSuffix. Algorithm The algorithm to discover new pattern using hash table is as following. Algorithm 2 (FindNewPattern) Input: Pattern Tree(PT), Hash Table(HT), New Entered Sequence(s_new) Output: Updated Hash Table(HT) 1: Delete old sequence as many as old_seqs times in OIT and then update DIT. 2: Input s_new into Hash Table in sequence and update OIT and DIT. pNext is null value. 3: foreach data D := s_new, in ascending order of nSuffix do 4: if nFreq of D >= min_sup then 5: Add D to PT 6: end if 7: foreach Data pre_D := s_new in descending order of nSuffix in Suffix of D do 8: if nFreq of pre_D >= min_sup then 9: Link D and pre_D, then add to PT 10: pNext of D := pre_D 11: if pNext of pre_D != null then 12: point the pattern in PT to the pattern begun by pNext of pre_D, then don't trace pNext 13: end if 14: end if 15: end for 16: end for

Fig. 5. Hash Table modified by FindNewPattern algorithm

Figure 5 describes hash table as an example of our algorithm. min_sup is equal to 2. Seq1= and Seq2= are entered previously, Seq3= is the newest entry. Left side in Figure 5 describes when Seq2 is entered, and right side describes hash table modified after pattern is analyzed. In the right side, the gray color part is the part that is modified. The process that modifies hash table is as follows. Seq2 is entered and nFreq, nSuffix are updated. We analysis processes in descending order of nSuffix among data in Seq3. Although , <e> have same nSuffix value, entered later is analyzed first. Because nFreq of is equal to 2, it becomes pattern. And search is terminated because there is no Suffix of .

RSP-DS: Real Time Sequential Pattern Analysis over Data Streams

105

Next is the turn of <e>. Because the nFreq of <e> is equal to 2, it also becomes a pattern. Then Suffix of <e> in each sequence is analyzed. That is, the nSuffix of data entered in Seq3 become standard between <> of Seq2 and of Seq3. But, , is entered just one time, so it can't be a pattern. Therefore, only <e> is discovered as pattern. Then, whose nSuffix is equal to 5, is analyzed. The nFreq is equal to 3, it become pattern. Suffix of is analyzed in each sequence. of Seq1, of Seq2, of Seq3 is analyzed according to nSuffix of data in Seq3. Because appears three times, it becomes a pattern. Because the nFreq of is 2 in the last data in Seq3, it become pattern. Suffix of in each sequence is then analyzed. of Seq1, <> of Seq2, <e,b,c> of Seq3 are analyzed according to nSuffix of data <e,b,c> in Seq3. Because appears two times and is linked , become patterns. Then the pNext of is not null but . So, becomes pattern. However, <e> appears just one time, so it can't be a pattern. 3.2 Pattern Deletion If patterns are added continuously, there are too many patterns in a pattern tree. Moreover, all of the sequences entered may become pattern in the worst case. So the patterns that don't appear well should be deleted in spite of addition of new patterns. Data Structure The way of deleting patterns can be adapted from the Buffer Replacement Policy[12]. The difference between Buffer Replacement Policy and deletion of pattern is that the number of Buffer items used in Buffer Replacement Policy is regular, but the number of patterns in Deletion of pattern isn't. That is, in case that very few patterns appear frequently, there are fewer patterns in the pattern tree. But, in case when there are many patterns that appear rarely, there are a lot of pattern in pattern tree. Therefore, if there are no more entered patterns over a regular period, it is possible to delete specific patterns. Comparing LRU(Least-Recently Used) Buffer Replacement Policy can be adapted for deletion method of RSP-DS. LRU policy processes when buffer is full, on the other hand, it is necessary to delete buffer when the scheduled time expires regularly in RSP-DS. LRU is not suitable for RSP-DS. There is another Buffer Replacement Policy, FIFO(First In, First Out) Queue that deletes the oldest block. But, It considers only time, not access frequency. We modify FIFO Queue algorithm to delete patterns easily. First, FIFO is generated by the time that pattern is entered. Then if that pattern is entered again, we input it in FIFO. We use pattern tree for access purpose, index of FIFO queue. Figure 6 describes modified pattern tree and FIFO structure.

Fig. 6. Pattern Tree with FIFO for deleting pattern

106

H.-S. Kim et al.

Algorithm The following is the algorithm to add pattern and delete pattern considering deletion of a pattern. Algorithm 3 (AddPattern) Input: Pattern Tree(PT), New Entered Sequence(s_new), FIFO Output: Null 1: if s_new PT then 2: Find s_new in FIFO, then move s_new to end of FIFO 3: else 4: Add s_new to PT, then add s_new and the entered time in FIFO 5: end if

∈

Fig. 7. Pattern Tree and FIFO modified by AddPattern()

Figure 7 describes the feature when pattern that already exists in pattern tree is entered. Simply, search the entered pattern in tree and input it to FIFO node again. Algorithm 4 (DeletePattern) Input: FIFO, pattern tree PT, Time Threshold t Output: Null

1: pat_set(Pattern Set) := Pattern over time t in FIFO 2: foreach pat := a pattern in pat_set do 3: if there is sub-pattern of pat in PT then 4: Delete pat and sub-pattern of pat in PT. If there is the pointer that points to sub-pattern, do not trace pointer. 5: else 6: Delete pat in PT 7: end if 8: end for

Fig. 8. Pattern Tree and FIFO modified by DeletePattern()

Figure 8 shows the process of deletion of a pattern. Even if is not frequent, it does not mean that is not frequent. Also, pattern that shares same

RSP-DS: Real Time Sequential Pattern Analysis over Data Streams

107

Prefix can be frequent. So, can be deleted, but not . If is not frequent, also and are not frequent. Therefore, when is deleted, its Suffix should be deleted. Two cases can be considered when a pattern should be deleted is a Suffix or a Prefix of other pattern. In the case of deletion of that is a Suffix of , in the tree and a pointer from is deleted. And, when deleting that is a Prefix, if some patterns are not frequent then all patterns that include that pattern are not frequent. So, and its child patterns can be deleted in pattern tree. Otherwise, if some of the child patterns have pointers that references another patterns, these patterns can be referenced from other patterns by its Suffix. So, these patterns are not deleted. Figure 9 shows an updated tree of Figure 2 when new sequence is inserted. These patterns that are not discovered from new pattern extraction process are excluded from pattern tree and newly discovered patterns are included in the pattern tree. Patterns in <<>> are newly included patterns that are not extracted from the previous tree.

Fig. 9. Pattern Tree resulted by RSP-DS

4 Performance Evaluation In this paper the presented algorithms have various performance measurements taken against four kind of variables. Next is the explanation of these variables. y y y

y

min_sup: A basis to estimate data frequency. In case of large min_sup, the calculation cost is reduced because the number of data is decreased. However, too small number of patterns makes it difficult to get appropriative results. dif_items: A kind of data stream. Many kinds of inserted data mean that there exist various data types. The value of this variable is changed by stream environment and user cannot modify it. new_seqs: The number of new sequence that will be inserted in hash table. To analyze fixed number of sequence, the last of new_seqs sequences are deleted from hash table whenever new_seqs sequences are inserted. A large size of new_seqs increases delay time and analyzing frequency. A small size of new_seqs decreases it, but calculation cost can grow up. old_seqs: The number of old sequences in hash table. This is the number of old sequences to be compared with new_seqs to analyze patterns. If old_seqs are larger than new_seqs then patterns are more easily classified whatever there is new_seqs. For example, let new_seqs is 3, old_seqs is 7 and min_sup is 6. Then data that appears after five times is decided as frequent data from new_seqs + old_seqs. At this time, if there is data D that is n times smaller than 3 exists in hash table, it is not selected to frequent data even if all of new three

108

y

H.-S. Kim et al.

sequences have D, since n+3 is lower than 6. Therefore, if data D' that satisfies n>=5 exists in the hash table, D is selected as frequent data no matter what the new sequence is. data_rate: The insertion ratio of sequence per time. If the basis of measurement is second and data_rate is 10, 10 sequences are inserted per second. This means that a unit of time_stamp is 1/10. In case when data_rate goes to excess, pattern analyzing speed cannot keep up with insertion speed. Therefore, if data_rate is lengthy, pattern analyzing frequency should be long to decrease the frequency.

Figure 10 is a building time comparison of MILE with data stream analysis that is divided by time window and RSP-DS tree. The horizontal axis represents data_rate per second and vertical axis represents average time of pattern analysis since first insertion of sequence. In the experiment, the value of dif_item, min_sup, new_seqs and old_seqs is 10. Table 1 lists update frequency of pattern tree through data_rate. In the case of MILE, tree should be built again whenever 10 sequences are inserted. Also, if the value of data_rate exceeds, update frequency is shortened and pattern calculation speed cannot keep up with data insertion speed. Therefore, processing time per sequence is extremely increased because of a lot of pending sequences for insertion in hash table. However, RSP-DS inserts new sequences in hash table, and compares with previous sequences, and then updates pattern tree. It demonstrates faster processing time than MILE.

Fig. 10. A comparison of RSP-DS and MILD Table 1. Update frequency of pattern tree with various value of data_rate Data_rate Update Frequency (sec)

10 1

15 2/3

20 1/2

25 2/5

30 1/3

35 2/7

Figure 11 is a performance result about RSP-DS with various values of min_sup. The value of dif_item, new_seqs, old_seqs is 10. min_sup is a ratio of new_seqs to have general value. As min_sup increases, processing cost is reduced and performance is increased by selecting frequent data from a small number of data. But, selecting too small number min_sup causes incorrect results, since it decreases the number of pattern being analyzed.

RSP-DS: Real Time Sequential Pattern Analysis over Data Streams

109

Fig. 11. A performance evaluation of min_sup

Figure 12 is a performance result as a result of variation of new_seqs. The value of dif_item, min_sup and old_seqs is 10. Table 2 shows the update frequency variation of new_seqs. If new_seqs increases, also update frequency is increased. So, increment of data_rate doesn't affect performance that significantly. However, it means that analysis performance decreases. Decreasing new_seqs reduces pattern analyzing time because it has a small number of sequences. Otherwise, high value of data_rate makes a short pattern analyzing frequency. So, pattern analyzing time is extremely increased because of low data insertion speed.

Fig. 12. A performance evaluation of new_seqs Table 2. Update frequency of pattern tree of data_rate and new_seqs (second) data_rate new_seqs=3

10 3/10

15 1/5

20 3/20

25 3/25

30 1/10

35 3/35

new_seqs=5 new_seqs=10

1/2 1

1/3 2/3

1/4 1/2

1/5 2/5

1/6 1/3

1/7 2/7

new_seqs=20

2

4/3

1

4/5

2/3

4/7

110

H.-S. Kim et al.

5 Conclusion Recently, data stream that has a new attribute and different from other data is being carefully studied. The pattern of data stream always changes. But, existing pattern analyzing algorithms have considered only single number of pattern analysis. This process has severe disadvantage that they repeat overall process again to analyze the patterns. The proposed RSP-DS method makes pattern tree by discovered pattern and patterns that gradually changes is analyzed at the branch of pattern tree in real time. RSP-DS is based on MILE that is a data stream pattern analyzing algorithm. But, the proposed method has better performance than MILE in environments with continuously pattern changes, and this is demonstrated by performance evaluation. This is achieved as MILE always creates a new tree whenever new sequences are inserted, but RSP-DS keeps current patterns of tree by editing them. And, experiments show that varying min_sup value and the number of sequences regulates performance of the proposed algorithm. Therefore, performance optimization can be done by manipulating these two variables. Hence, a research of pattern analysis of data stream whose frequency changes continuously should be done for RSP-DS, as currently it only considers fixed frequency. And, in environment where patterns rarely change, RSP-DS algorithm processes unnecessary operations. So, study of discovering pattern variation is required which is left as a future work.

References [1] Pei, J., Han, J., Mortazavi-Asl, B., Wang, J., Pinto, H., Chen, Q., Dayal, U., Hsu, M.C.: Mining sequential patterns by pattern-growth: The prefixspan approach. IEEE Trans. Knowl. Data Eng. 16(11), 1424–1440 (2004) [2] Chen, G., Wu, X., Zhu, X.: Sequential Pattern Mining in Multiple Streams In: ICDM, pp. 585–588 (2005) [3] Babcock, B., Babu, S., Datar, M., Motwani, R., Widom, J.: Models and Issues in Data Stream Systems. In: PODS, pp. 1–16 (2002) [4] Srikant, R., Agrawal, R.: Mining sequential patterns: Generalizations and performance improvements. In: Apers, P.M.G., Bouzeghoub, M., Gardarin, G. (eds.) EDBT 1996. LNCS, vol. 1057, pp. 3–17, Springer, Heidelberg (1996) [5] Li, H.F., Lee, S.Y., Shan, M.K.: On Mining Webclick Streams for Path Traversal Patterns. WWW, pp. 404–405 (2004) [6] Li, H.F., Lee, S.Y., Shan, M.K.: DSM-TKP: Mining Top-K Path Traversal Patterns over Web Click-Streams Web Intelligence, pp. 326–329 (2005) [7] Das, G., Lin, K.I., Mannila, H., Renganathan, G., Smyth, P.: Rule Discovery from Time Series, KDD, pp. 16–22 (1998) [8] Oates, T., Cohen, P.R.: Searching for Structure in Multiple Streams of Data. In: ICML, pp. 346–354 (1996) [9] Li, H.F., Lee, S.Y., Shan, M.K.: On Mining WebClick Streams for Path Traversal Patterns, WWW, pp. 404–405 (2004) [10] Gaber, M.M., Krishnaswamy, S., Zaslavsky, A.: Ubiquitous Data Stream Mining. In: Dai, H., Srikant, R., Zhang, C. (eds.) PAKDD 2004. LNCS (LNAI), vol. 3056, Springer, Heidelberg (2004) [11] Zhao, Q., Bhowmick, S.S.: Sequential Pattern Mining: A Survey, Technical Report Centre for Advanced Information Systems, School of Computer Engineering, Nanyang Technological University, Singapore (2003)

Investigative Queries in Sensor Networks Madhan K. Vairamuthu, Sudarsanan Nesamony, Maria E. Orlowska, and Shazia W. Sadiq School of ITEE, The University of Queensland, Australia {madhan,sudar,maria,shazia}@itee.uq.edu.au

Abstract. Sensor networks have widespread range of applications these days. New computing locations, new needs and fresh applications have instigated new problems and exciting challenges. The view of sensor network as a distributed database and the sensor node as a table is being widely accepted by the database community. There now exists wide variety of data models and query processing methods to acquire data from sensor networks. This paper identifies a new class of query, named the investigative query, owing to needs of few special applications. Classifying sensor network query processing from communication perspective, defining the new class of query and architecture requirements for processing the same, this paper gives an application of those queries in urban water distribution management and concludes with future directions. Keywords: Investigative Query Processing, Sensor Networks.

1 Introduction Sensor network applications have brought a phenomenal change in how this world views Information Technology as a field of science. In the event of taking ‘computing’ from labs’ desks to farms & roads, computer science has made huge sacrifices to cope up with energy constraints in the new little computing machines. The leap is huge, the paradigm shift to real world is brilliant but in the process we have lost much of what we used to call the power of computing. Low power, low communication range, micro storage, tiny processor, tiny operating system and tiny database are the typical characteristics of today’s sensor node in a typical sensor network. Sensor networks have thus taken us on a beautiful journey on a time machine, few decades backwards in terms of computing. The same old problems of memory management, communication, computations on a low capability processor and many more such classical problems have been revisited in the recent sensor network literature. On the other side, emergence of sensor networks in new domains has excited some fresh applications and new problems to the research community. They actually have injected fresh blood into the veins of distributed computing. Distributed data management and query processing were enlivened by the advent of these distributed sensors. The sensor network research community has two major views of sensor K.C. Chang et al. (Eds.): APWeb/WAIM 2007 Ws, LNCS 4537, pp. 111–121, 2007. © Springer-Verlag Berlin Heidelberg 2007

112

M.K. Vairamuthu et al.

networks. One half of the community views sensor nodes as tiny computers networked together in a huge space. Another half views the tiny sensors as small relational tables and the entire network as an active database. A huge portion of research is devoted to processing of sensor network queries. This paper observes a rise of a new class of queries in sensor networks owing to the demand of some new applications. We call them investigative queries. Investigative queries are active recursive queries that probe into a network seeking some application data. They are executed in hierarchical fashion. The data collected by one level of this query will decide on the execution of the next level of the same query. This paper aims at defining investigative queries after classifying the list of queries available in sensor network literature. Dissemination of investigative queries into the network requires some special communication architecture to provide the hierarchy levels needed for the application. The requirements are discussed in detail. We provide a strong motivating application to demonstrate the need of investigative queries. The paper is organised into five major sections. Section 2 aims at classifying the sensor network query routing protocols and provides different types of queries that exists in sensor network literature. In section 3 a sensor network application is described in detail where a new type of query called the investigative query is introduced. Formal definition of investigative queries is provided in section 4, which is the main contribution of this paper. The advantages of investigative queries over traditional methods are discussed along with other potential applications of sensor networks in the same section.

2 Classification of Queries in Sensor Networks In this section we classify query processing in sensor networks in terms of their communication architecture and then by the data processing methodology. Query processing in sensor networks can be broadly classified based on their query dissemination and data gathering approaches into four major classes, • • • •

Data Flooding Tree Path routing Multi Path routing Clustered routing

Each of these approaches can be sub classified based on their data processing approaches. • •

Centralised In-network

In order to achieve this classification, we make the following reasonable assumptions. • •

Existence of a base station that can issue queries and collect and store results Capability of sensors to sense, store, transmit and route data.

Investigative Queries in Sensor Networks

113

2.1 Data Flooding Data flooding is the simplest of all the approaches, yet powerful. A query issued by base station is flooded into the network by, every sensor node transmitting the query to every node in its neighbourhood until the query is propagated to the entire network. The result is propagated in the similar fashion outwards [1-2]. The advantage of such a technique is its reliability, but the disadvantage being overall energy consumption of the network even for processing simple queries. 2.2 Tree Path Routing Tree path routing is another simple technique where a tree is constructed out of the available dense network and the tree is maintained throughout for disseminating queries and collecting results [3-4]. Contrasting to the data flooding approach, this approach suffers from single node failure. Considering the cost of constructing a new tree, the maintenance of the tree becomes inefficient. The major advantage of this method is that it conserves over all network energy. 2.3 Multi Path Routing Multi path routing can be considered as an approach in between data flooding and tree path routing [5-6]. The challenge in this approach has been to join the data from multiple paths and eliminating redundancy from multiple paths. 2.4 Clustered Routing Numerous clustering algorithms [7-9] group nodes that are spatially close into clusters. The advantage of clustering is that it provides a hierarchical structure for executing queries and the disadvantage being the cost associated with clustering and electing cluster heads [8]. 2.5 Centralised vs. In-Network Processing Centralised data processing is a very common approach where using any of the four classified approaches, query is propagated through the network and result from every node is sent back to the base station in a central location and the data is processed centrally to obtain result for given query. The advantage of this method is that once data is collected, multiple queries can be run over the available data and the main disadvantage being energy spent by individual nodes to transmit their data and route their neighbours data to the central station [10]. In-network processing is an alternative technique that minimises a node’s energy consumption by performing computation at node level, combine the results and transmit just the aggregate result to the base station. This approach requires the nodes to have some basic data processing capabilities [10].

114

M.K. Vairamuthu et al.

2.6 Query Types in Sensor Networks Numerous types of queries have been introduced in sensor network literature. Simple aquisitional queries are used for acquiring data from specific nodes in a sensor network. They are very similar to simple distributed queries, which perform similar action. These queries are mainly executed with a tree path routing communication architecture or the flooding architecture [11]. Acquiring the average value of nodes or set of nodes in a network is a very common query in sensor networks. Such queries are termed as aggregate queries and have a wide variety of applications [12]. They are executed on most of the architectures specified. Tree path routing with in-network processing has been the most popular architecture to run these aggregate queries. Queries that are targeted at nodes with a certain range of values fall under the class of range queries[13]. Range queries are very handy in temperature and pressure monitoring applications. Clustering nodes based on the their location and previous data range has proven to be suitable for these kind of queries. Window queries are very specific to spatial data. A user specified window selects the window-contained nodes alone to respond to the given query. These queries are very popular in numerous applications [14]. Tree-path and multi-path routing is proven to be efficient than flooding in many applications. Considering the poor reliability of weak sensor nodes, the notion of probability is introduced in to queries [15]. Probabilistic queries are not specific to any architecture. The probability takes into account the error in results owing to frequent node failure. Stream queries [16] and active queries [17] are being used in few real time monitoring applications, which require real time action. 2.7 Need for Advanced Query Types Advanced applications of sensor networks demand architecture and means to support advanced queries. Queries issued by a central base station to identify location of a leak in water distribution pipes’ sensors and queries issued to smart buildings to identify unoccupied rooms where the electricity usage is high are new to the database community. The demand of such applications brings in the need for new query types. The following section discusses one such application in detail.

3 A Sensor Network Application with Advanced Queries In [18] we identified the potential benefits of deploying sensor networks for urban water management especially to identify leaks in the distribution pipes. We had explained the need to segment the network in order to hierarchically query the sensors for existence of a leak. In the following sections, we present a water network model, a sensor network model and communication model similar to [18]. The need for investigative queries is discussed at end of this section. 3.1 Urban Water Distribution Network Model In this section first we present an abstraction of the real pipe network by the following mathematical model. We begin with introduction of essential notation.

Investigative Queries in Sensor Networks

115

Let the water distribution network W = (S, C, P) consists of a set of supply sources S = {s1 , s 2 , ..., ss } , a set of consumption units C = {c1 , c 2 , ..., cq } , and a set of connected pipes P = {p1 , p 2 , ..., p m } .

Fig. 1. A Water distribution network W

3.2 Sensor Network Model

We define three types of sensor nodes to be deployed for urban water management. • • •

Consumption Sensors Distribution Sensors Fragmentation Sensors

A consumption sensor is placed in front of every consumption unit for the purpose of measuring the total water consumption of a consumption unit over a monitored time period. Without any loss of specification precision, we use the same notation for these sensors as for corresponding water network consumption units, we call them ci , where i = (1, 2, …, q). A distribution sensor is placed at entry/exit of every pipe segment in the water network. Essentially every pipe segment would have two distribution sensor nodes, one on each end. These distribution nodes monitor the total inflow and outflow of that particular pipe over a monitored time period. We thus extend our collection of sensors deployed by defining distribution nodes 2 D = {d11 , d12 , d12 , d 22 , ..., d1m ,d m } where d1j and d 2j denotes the pair of sensor nodes placed one each at both the ends of the pipe segment p j ∈ P in W. Fragmentation sensors are positioned at certain position in the network such that they fragment the network into required number of connected segments and required levels of hierarchy. This hierarchical segmentation is abstracted in section 6. We now have a third set of sensors that fragment the network as F = { f1, f 2 ,..., f h } where a fragmentation sensor fi monitors the total inflow and outflow of one or two segments over a monitored time period.

116

M.K. Vairamuthu et al.

Fig. 2. SN(W) for w in figure 1

With this arrangement we have a mundane way of identifying leaks in every pipe by querying the distribution sensor at the end of every pipe segment. We have a set of points from which water flows into the whole network of pipes. We also will have the water consumption details at every consumption point in the whole network for a specific period of time. If the amount of water flowing into the network is more than the total consumption of water within the network, then it can be concluded that leak(s) are present within the network, based on various practical and reasonable assumptions discussed in [18]. Thus, if the presence of leaks is detected, then every single pipe within the network will have to be examined for a leak. In practice and in theory, this is a very ineffective and uninteresting method to solve the problem. In real world, the number of pipe segments is very high in number and the percentage of pipes containing leaks is very small. Hence, we seek a better method that does not involve unnecessary querying of all the nodes. In [18], we proposed a technique by which, we divide the network into various partitions and perform operations at the partition level rather than the entire network. Conceptually, a fragmentation of the network is constructed with many hierarchical levels. Initially, the entire network is broken down into a set of connected fragments, with each segment containing a portion of the network. The collection of all the segments will comprise the initial network. The segmentation nodes are placed at the end points of the constructed segments, which measure the net inflow of the water into every segment. This is compared against the actual consumption of water by the end units that are located within that particular segment. Thus, instead of querying every pipe in the current segment, it is sufficient to pass on the query to the subsegments that were created. The levels of hierarchy that is needed for the segmentation of the entire network will depend on the application requirement and the characteristics. Eventually, the lowest level will be pipe level segments. We now have segment level distribution nodes apart from the pipe level distribution nodes. A segment level distribution node monitors the inflow and outflow volume of water pertaining to a segment over a monitored time period. It is to be noted that such a distribution node could belong to two different segments and still has capability to maintain data pertaining to both the segments.

Investigative Queries in Sensor Networks

117

3.3 Communication Model

The central processing station should be aware of the readings of consumption sensors over the entire network, as the fundamental requirement of the application is to measure the water intake by every consumption unit. The readings are taken for a period of time and considering the scope of the network size, the ideal communication method will be to include gateway nodes that are powerful enough to reach the base station through wireless signals and the consumption sensor nodes should be able to communicate with the gateway nodes. The communication architecture of the distribution nodes can be the same as that of the consumption nodes. For the current problem, we assume that the communication method is reliable and functional. Taking into account the scope of the network span, it is very reasonable to assume the presence of gateway nodes to facilitate communication amongst the various types of sensor nodes that prevail in the network. 3.4 Fragmentation Model

Let F0 ⊂ F be the level zero fragmentation sensors that defines the entire network. Similarly ∀ i=1..l Fi ⊂ F denotes a set of subsets of ith level fragmentation sensors, where l denotes the number of levels of segmentation. There by, Fi = {Fi1,Fi2,...,Fip } where p denotes the number of fragments in level i. So, a fragment of network W can be obtained by cutting the network W at the position of ∀ i f i ⊂ Fk ⊆ F for any subset of fragmentation sensors that define a fragment. In this paper, we presume the existence of a good fragmentation of the network that satisfies the constraints associated with it. We proceed with the result of [] that has provided a fragmentation of the given network. We use this fragmentation as an input to the Investigative queries that have been described earlier. For facilitating this purpose, the fragmentation of the network has to be abstracted. We provide the abstraction of the hierarchical fragmentation as a tree. Let A be the entire fragment representing W defined by fragmentation sensors in F0. Let A1 , A 2 , ..., A p be the first level fragments of A into sub-fragments of A. Each Ai is a fragment of W defined by the fragmentation sensors F1i. We obtain the segments in the next levels in similar fashion. Therefore, Ai = A denotes that the

U

i=1..p

union of fragments, in each level with the corresponding fragmentation sensors, will be the entire fragment representing W. Let T be a tree representing the fragmentation of W as shown in figure 3. This tree is the result of the network fragmentation process that is detailed in [18]. This representation will capture the level of positioning and hierarchy of the individual segments in the network. The tree structure suits the best for manipulating hierarchical data. The investigative queries will operate on this structure as they execute themselves over a recursive hierarchy.

118

M.K. Vairamuthu et al.

Fig. 3. Fragmentation tree T

The tree T is rooted at a node that denotes the topmost level of hierarchy of the segmentation, which is the entire network itself. All the leaf nodes of T represent the lowest level of hierarchy of segmentation, which will be all the individual pipe segments present in the whole network. The conceptual meaning of the leaf nodes carries significance as the investigative queries, when detected mismatch between the net inflow and the total consumption in a segment, execute recursively within the subsequent level of sub segments and finally terminate at one or more leaf nodes meaning that the pipe segments denoted by those leaf nodes contain the leaks. The internal nodes of the tree will symbolise the intermediate levels of segments that are neither the entire network nor the individual pipe segments. Since T is a rooted tree, we can also define the notion of a child node of a node in the tree. The child node of a given node will be any node that is directly connected to the given node, which has got the path distance from the root node that is one greater than the path distance between the root node and the given node. Conceptually, a child node denotes a sub-segment or one component of a partition of the given segment represented by the given node. 3.5 Need for Investigative Queries in UWD Management

In the described scenario, querying the sensor nodes for identifying location of leak will be a periodical operation. One way of locating leaks would be to query every distribution sensor and consumption sensor in the network. With Investigative queries we minimise the number of distribution sensors being queried. The fragmentation of the water network gives us a hierarchy as described in the previous section. This hierarchy provides efficiency in processing the query by reducing the number of distribution sensors that need to be queried. The following section formally defines investigative queries in general.

4 Investigative Queries 4.1 Definition

Let Q(s) be an investigative query issued over a segment of interest, in the given network, denoted by node s of segmentation tree T, such as the one mentioned in section 4.4. Now, Q(s) can be defined as follows.

Investigative Queries in Sensor Networks

119

⎧s leaf (s) = true ∧ δ (s) > θ ⎪ Q(s) = ⎨Q(sc )∀sc ∈ child(s) leaf (sc ) = false ∧ δ (s) > θ ⎪ null otherwise ⎩ The segment of interest denoted by node s is usually the root of T. But in general, s can be any node in T denoting any segment that needs to be investigated. Here, leaf(s), a Boolean function, will return true if s is a leaf node or false otherwise. Function child(s) will return a set of nodes from T containing the children nodes of s. The investigative function δ (s) and threshold θ are provided by the application on which these queries are run. 4.2 Investigative Queries in Urban Water Management

In the urban water management scenario δ (s) balances the inflow volume of water into segment denoted by s against the consumption volume in the same segment. This function is explained in detail in [ref]. The threshold limit for δ (s) is given by θ . The value of θ is determined by the organisation that controls water distribution. The query returns the nodes, corresponding to the pipe segments that contain the leak or returns null if there exists no leak in the network. The efficiency benefits of using investigative queries over simple aquisitional queries are explained in the following section. 4.3 Advantage over Traditional Queries

Let m denote the number of pipes in the water network thus 2m is the number of distribution nodes installed. Let l denote the level of fragmentation and maximum number of fragmentation sensors in any level be denoted by fs <<2m. To Identify leak using a simple acquisitional query, we need to query 2m distribution nodes. With an Investigative Query, the number of fragmentation nodes and distribution nodes that need to be queried in the worst case, where every level has fs nodes, is given by lfs+2 which includes the two distribution nodes in the pipe that has the leak. So, for a 75,000-pipe network with 150,000 distribution nodes, we would need 150,000 readings from distribution nodes to locate a leak with an acquisitional query. Whereas an investigative query, on a 10-level fragmentation hierarchy with value of fs being 100, requires 1002 readings to locate a leak. 4.4 Other Possible Applications of Investigative Queries

Investigative queries can be applied in many applications where the data is needs to be sought in a hierarchical fashion. The same leak detection can be used in pipes applied for petroleum industries. Supply chain management with RFIDs can be a potential application for Investigative queries. Item dislocation or path of an item can be queried in such applications. A group of buildings, say a University, where there

120

M.K. Vairamuthu et al.

exists a natural hierarchy as streets, buildings, floors and rooms where sensors are deployed to sense occupancy, sound, temperature and light exposure. We can have a series of Investigative queries here such as locating rooms in buildings with high electricity usage with no occupants. Locating toilets in buildings with water dripping taps will be another application of investigative querying. So, a wide range of applications exists to deploy investigative queries. Different application needs different architecture to support investigative querying.

5 Conclusion In this paper, we classified the query processing strategies and the types of queries in sensor network literature. We identified newly evolving query types due to deployment of sensors in diverse applications. Urban water distribution with sensor networks was discussed as a motivating application where investigative queries are introduced. The advantages of investigative queries were discussed after formally defining the new class of queries. Few applications that can deploy investigative queries were also listed for further research and investigation under this topic.

References 1. Maróti, M.: Directed flood-routing framework for wireless sensor networks. In: Proceedings of the 5th ACM/IFIP/USENIX international Conference on Middleware, Toronto, Canada, October 18 - 22, 2004. Middleware Conference, vol. 78, pp. 99–114. Springer, New York (2004) 2. Mehta, S., Lee, J., Kim, J.: IS-MAC based flooding protocol for sensor networks. In: Proceedings of the 2nd ACM international Workshop on Performance Evaluation of Wireless Ad Hoc, Sensor, and Ubiquitous Networks, PE-WASUN ’05, Montreal, Quebec, Canada, October 10 - 13, 2005, pp. 79–83. ACM Press, New York (2005) 3. Lian, J., Chen, L., Naik, K., Özsu, M.T., Agnew, G.: Localized routing trees for query processing in sensor networks. In: Proceedings of the 14th ACM international Conference on information and Knowledge Management. CIKM ’05, Bremen, Germany, October 31 November 05, 2005, pp. 259–260. ACM Press, New York (2005) 4. Moh, M., Dumont, M., Moh, T., Hamada, T., Su, C.: Brief announcement: evaluation of tree-based data gathering algorithms for wireless sensor networks. In: Proceedings of the Twenty-Fourth Annual ACM Symposium on Principles of Distributed Computing. PODC ’05, Las Vegas, NV, USA, July 17 - 20, 2005, p. 239. ACM Press, New York, (2005) 5. Zhu, J., Hung, K., Bensaou, B.: Tradeoff between network lifetime and fair rate allocation in wireless sensor networks with multi-path routing. In: Proceedings of the 9th ACM international Symposium on Modeling Analysis and Simulation of Wireless and Mobile Systems. MSWiM ’06, Terromolinos, Spain, October 02-06, 2006, pp. 301–308. ACM Press, New York (2006) 6. Dong, Q., Banerjee, S., Adler, M., Misra, A.: Minimum energy reliable paths using unreliable wireless links. In: Proceedings of the 6th ACM international Symposium on Mobile Ad Hoc Networking and Computing, Urbana-Champaign, IL, USA, May 25-27, 2005, pp. 449–459. ACM Press, New York (2005)

Investigative Queries in Sensor Networks

121

7. Arboleda, C.L.M., Nasser, N.: Cluster-based routing protocol for mobile sensor networks. In: Proceedings of the 3rd international Conference on Quality of Service in Heterogeneous Wired/Wireless Networks. QShine ’06, Waterloo, Ontario, Canada, August 07-09, 2006, vol. 191, p. 24. ACM Press, New York (2006) 8. Moussaoui, O., Naïmi, M.: A distributed energy aware routing protocol for wireless sensor networks. In: Proceedings of the 2nd ACM international Workshop on Performance Evaluation of Wireless Ad Hoc, Sensor, and Ubiquitous Networks. PE-WASUN ’05, Montreal, Quebec, Canada, October 10-13, 2005, pp. 34–40. ACM Press, New York (2005) 9. Luo, H., Luo, J., Liu, Y.: Energy efficient routing with adaptive data fusion in sensor networks. In: Proceedings of the 2005 Joint Workshop on Foundations of Mobile Computing. DIALM-POMC ’05, Cologne, Germany, September 02-02, 2005, pp. 80–88. ACM Press, New York (2005) 10. Madden, S.R., Franklin, M.J., Hellerstein, J.M., Hong, W.: TinyDB: an acquisitional query processing system for sensor networks. ACM Trans. Database Syst. 30(1), 122–173 (March 2005) 11. Madden, S., Franklin, M.J., Hellerstein, J.M., Hong, W.: The design of an acquisitional query processor for sensor networks. In: Proceedings of the 2003 ACM SIGMOD international Conference on Management of Data. SIGMOD ’03, San Diego, California, June 09-12, 2003, pp. 491–502. ACM Press, New York (2003) 12. Sharifzadeh, M., Shahabi, C.: Supporting spatial aggregation in sensor network databases. In: Proceedings of the 12th Annual ACM international Workshop on Geographic information Systems. GIS ’04, Washington DC, USA, November 12-13, 2004, pp. 166– 175. ACM Press, New York (2004) 13. Kalpakis, K., Puttagunta, V., Namjoshi, P.: Accuracy vs. lifetime: linear sketches for approximate aggregate range queries in sensor networks. In: Proceedings of the 2004 Joint Workshop on Foundations of Mobile Computing. DIALM-POMC ’04, Philadelphia, PA, USA, October 01-01, 2004, pp. 67–74. ACM Press, New York (2004) 14. Coman, A., Nascimento, M.A., Sander, J.: A framework for spatio-temporal query processing over wireless sensor networks. In: Proceeedings of the 1st international Workshop on Data Management For Sensor Networks: in Conjunction with VLDB 2004. DMSN ’04, Toronto, Canada, August 30-30, 2004. DMSN ’04, vol. 72, pp. 104–110. ACM Press, New York (2004) 15. Lam, K., Cheng, R., Liang, B., Chau, J.: Sensor node selection for execution of continuous probabilistic queries in wireless sensor networks. In: Proceedings of the ACM 2nd international Workshop on Video Surveillance &Amp; Sensor Networks. VSSN ’04, New York, NY, USA, October 15-15, 2004, pp. 63–71. ACM Press, New York (2004) 16. Srivastava, U., Munagala, K., Widom, J.: Operator placement for in-network stream query processing. In: Proceedings of the Twenty-Fourth ACM SIGMOD-SIGACT-SIGART Symposium on Principles of Database Systems. PODS ’05, Baltimore, Maryland, June 1315, 2005, pp. 250–258. ACM Press, New York (2005) 17. Zoumboulakis, M., Roussos, G., Poulovassilis, A.: Active rules for wireless networks of sensors & actuators. In: Proceedings of the 2nd international Conference on Embedded Networked Sensor Systems, SenSys ’04, Baltimore, MD, USA, November 03-05, 2004, pp. 263–264. ACM Press, New York (2004) 18. Nesamony, S., et al.: On Sensor Network Segmentation for Urban Water Distribution Monitoring. In: Zhou, X., Li, J., Shen, H.T., Kitsuregawa, M., Zhang, Y. (eds.) APWeb 2006. LNCS, vol. 3841, Springer, Heidelberg (2006)

A Taxonomy-Based Approach for Constructing Semantics-Based Super-Peer Networks Baiyou Qiao, Guoren Wang, and Kexin Xie School of Information Science and Engineering, Northeastern University, Shenyang, 110004, China {qiaobaiyou, wangguoren}@ise.neu.edu.cn

Abstract. Clustering a peer-to-peer (P2P) network into distinct semantic clusters can efficiently improve the search efficiency and enhance scalability of the network. This paper considers P2P systems in which peers employ taxonomy hierarchy to describe the contents of their objects, and presents a taxonomy-based approach for constructing semantics-based super-peer networks. By dynamically clustering peers in taxonomy-based semantic space based on the semantics of their data and organizing the clusters into semantic routing overlays, an efficient query-routing algorithm can be used among these clusters. Preliminary evaluation indicates that our approach achieves a competitive trade-off between search latencies and overheads, and loadbalancing among super-peers is well maintained.

1 Introduction A super-peer network is a hierarchy structure formed by adding super-peer nodes to an unstructured P2P system. It combine elements of both pure and hybrid P2P systems, it has the potential to combine the efficiency of a centralized search with autonomy, load balancing, robustness to attacks and at least semantic interoperability provided by distributed search. Therefore super-peer networks, such as KaZaA [7] and Gnutella’s new version [1], have been one of the most popular network applications. Taxonomy hierarchy is widely used in the real world, for its well conceptual hierarchy structure and supporting for logical reasoning. At present there are many distributed data sources which employ taxonomy hierarchy to describe the contents of their objects, such as PC files, web directories, and digital books and so on. And how can we efficiently exchange and share information among these large-scale distributed data sources which employ a domain taxonomy hierarchy to describe the contents of their objects? An effective solution is to construct a semantics-based super-peer network by taking advantage of semantic information of taxonomy hierarchy and benefits of super-peer networks. Paper [12] proposes a simple method based on unstructured P2P networks, i.e. peers are classified into concepts in the taxonomy hierarchy according to the content of their objects, and each semantic cluster is associated with a concept. This static clustering approach, however, for a super-peer network, can result in severe load imbalance among super-peers, thus K.C. Chang et al. (Eds.): APWeb/WAIM 2007 Ws, LNCS 4537, pp. 122–134, 2007. © Springer-Verlag Berlin Heidelberg 2007

A TBA for Constructing Semantics-Based Super-Peer Networks

123

inevitably reducing performance of the system. Paper [15] proposes an approach based on DHT, hashing the whole taxonomy hierarchy to super-peers, and super-peers are connected according to CHORD protocol. However, this approach results in loose semantics of data within a super-peer and makes the communications between superpeers increasing, and thus affects network performance. Other approaches, such as Edutella [6], Hypercup [16], etc. do not consider load-balancing among super-peers. Here this paper proposes a Taxonomy-based Approach (TBA), which considers the characteristics of data semantic space consisting of taxonomy hierarchy, dynamically clusters peers into many semantic clusters based on the semantics of their data and load, and organizes the clusters into semantic routing overlays. Each semantic cluster is composed of data and peers in its semantic sub-space and managed by a super-peer, and each cluster is only responsible for answering queries in its semantic sub-space. Compared to traditional super-peer networks, the approach employs a source locating strategy based on taxonomy hierarchy, and send queries only to the semantic clusters that possibly have results. Thus, peers involved and messages to send are reduced and network performance is greatly enhanced as a result. Preliminary evaluation shows that the approach achieves a competitive trade-off between search latencies and overheads, load-balancing among clusters and data semantics within a cluster are all well maintained. The remainder of the paper is organized as follows. System model is provided in section 2, section 3 describes the approach of constructing semantic based super-peer networks in detail, and section 4 gives a search and routing algorithm. Preliminary performance evaluation is presented in section 5. Related works are presented in section 6. Finally we conclude this paper.

2 System Model 2.1 Definition of Taxonomy Hierarchy and Encoding Taxonomy hierarchy is a model effectively describing data semantics and used widely in various fields of computer science, which has good conceptual hierarchy structure and support for logical reasoning. In the paper, we suppose that there is a common taxonomy hierarchy, and all data are classified and organized according to it. The taxonomy hierarchy represents the semantic space of data on the network, and hence a sub-taxonomy hierarchy stands for a semantic sub-space. A taxonomy hierarchy can be expressed with a tree, called a taxonomy tree, defined as follows. Definition 1. A taxonomy tree T=, where N is a set of all nodes, each node represents a concept, E is the set of directed edges, and an edge represents ISA relationship. Nodes on the tree can have arbitrary fan-outs and leaf nodes can be on different levels. Each node represents a concept. A concept in a higher layer is more general and contains more children concepts, and a concept in a lower layer is part of those in a higher layer. Each node has a unique ID, and the ID encoding is as follows. The ID of the root node of the taxonomy tree is prescribed to be 0, and the ID of the parent node is the prefix of its children nodes. Fig.1 is an example for encoding a taxonomy tree. The ID of the root node is 0, and the IDs of its children are 0.1, 0.2, and 0.3. IDs of other nodes are also shown in Fig.1.

124

B. Qiao, G. Wang, and K. Xie

The advantage of the encoding is that: when the IDs of two nodes are given, it is very easy to judge their relationship. When one ID is a prefix of another, the two nodes have the ascendant-descendant relationship. The encoding is used later. Cluster1

0 0.1

0.2

Cluster2

0.3 Peer

0.1.1 0.1.2 0.1.3 0.2.1 0.2.2 0.2.3 0.3.1 0.3.2 0.3.3

Cluster3

Super-peer Cluster4

Fig. 1. An example of encoding a taxonomy Fig. 2. The semantics-based tree network model

super-peer

2.2 Semantics-Based Super-Peer Network Model A semantics-based super-peer network is a hierarchy structure consisting of two layers, the bottom layer is composed of peers and the top layer is composed of superpeers, super-peers are connected into an overlay structure according to their semantics. A set of peers with their super-peer together forms a semantic cluster. Data are stored on peers with data index and routing information of neighbor super-peers stored on super-peers. Intra semantic cluster data communication takes place via direct peer-to-peer links; inter semantic clusters communication takes places via links between super-peers. Each semantic cluster uses a common taxonomy tree to describe its data semantics, and takes charge of a semantic sub-space, comprised of one or multi sub-trees of the taxonomy tree. If the data on each peer are classified into n nodes of the taxonomy sub-trees, there are n index items, and all index items are organized and stored in the form of one or multi sub-trees. Each cluster is only responsible for answering queries in its semantic sub-space. Each query is associated with a node of the taxonomy tree and the node express semantics of the query. The query is routed according to its semantics, so the peers involved and messages to send are reduced and network performance is greatly enhanced. Fig. 2 is the model of semantics-based super-peer network.

3 Approach for Constructing Semantics-Based Super-Peer Networks In this section the approach for constructing semantics-based super-peer network (TBA) is described in detail from four aspects. 3.1 Constructing Strategy We apply a partitioning strategy of the semantic space to construct the network, which a preset maximal cluster load size M is used to determine the clusters ranges. The chief idea is: suppose that there is one semantic cluster in the network initially, whose semantic space is the whole taxonomy tree; the load size of the cluster

A TBA for Constructing Semantics-Based Super-Peer Networks

125

increases with peers joining, if the load size of the cluster exceeds M, the cluster will automatically partition its semantic sub-space according to the data semantics and the load size, and a new semantic cluster is generated from the original one. The new semantic cluster selects appropriate semantic clusters as its neighbors, and answers queries in its semantic sub-space. In the way each cluster continuously partitions its semantic sub-space and a semantics-based adaptive super-peer network forms. The advantage of the approach is that the load size within a cluster is less than the maximal load size M, and hence search efficiency of the system is guaranteed. Therefore, the approach is adaptive to difference of density of peers in the semantic space, and ensures stability and adaptability of the system. 3.2 Clustering Algorithm Based on Taxonomy Hierarchy In a super-peer network, any super-peer, as an index server, takes charge of vast work such as maintaining index data, answering and routing queries, etc., and thus the capability of super-peers decides the network performance. Therefore, we regard the load of a super-peer as the load of the cluster, which is the important basis of clustering. Since maintenance and search take a super-peer a lot of time, the load of a super-peer can be measured by the number of index items. Thus we can express a semantic cluster with a weighted taxonomy sub-tree, the weight of a node representing the number of index items, and topological relationship between nodes represents semantic relationship between the corresponding index items. Now clustering problem can be regarded as partitioning a set of weighted taxonomy subtrees. When clustering, we should consider not only the semantics between index items but also the load-balancing among super-peers, and in fact, it is a tradeoff between load-balancing among clusters and semantics within a cluster. Aimed at the problem, we propose a Self-organized Semantic Clustering Algorithm, i.e. LBFCA (Load Balance First Semantic Cluster Algorithm) in paper [19], which achieves a good trade-off between data semantics within a cluster and load-balancing among clusters. LBFCA algorithm obeys the following three rules: 1. If the original cluster is composed of a sub-tree, the new cluster generated consists of one or multi sub-trees of a node on the original tree, which makes the difference of load sizes between the two clusters minimal; 2. If the original cluster is composed of multi sub-trees, the sub-trees are split into two groups, either of which forms a cluster, which makes the difference of load sizes between the two clusters minimal; 3. If the original cluster is composed of one node on the taxonomy tree, the new cluster consists of the same node, and the two clusters share the load of the original one. As shown in Fig. 3, SC0 is a semantic cluster, which can be expressed with a weighted taxonomy tree, where a weight of a node represents the number of index items of the corresponding concept and the sum of weights of all nodes is the load size of the cluster SC0, i.e. 120. If we assume that maximal cluster load size M is 120, the load size of SC0 reached M, and SC0 should automatically execute clustering algorithm, after applying LBFCA, a new cluster SC8 is produced. As shown in Fig. 4,

126

B. Qiao, G. Wang, and K. Xie

SC0 is composed of a sub-tree whose root ID is 0, and SC8 is composed of two subtrees whose root IDs are 0.2 and 0.3 respectively. Detailed description of LBFCA is in paper [19]. 0 0.1 15

5

0.2 8

SC0(0000)

SC0 (0000)

5

SC8 (1000)

0

0.3 6

13 8 18 10 9 6 7 6 9 0.1.1 0.1.2 0.1.3 0.2.1 0.2.2 0.2.3 0.3.1 0.3.2 0.3.3

Fig. 3. An example of a semantic cluster

15 0.1

13 0.1.1

8 18 0.1.2 0.1.3

0.2

0.3

8

6

10 9 6 7 6 9 0.2.1 0.2.2 0.2.3 0.3.1 0.3.2 0.3.3

Fig. 4. The clustering result by LBFCA

3.3 Cluster Encoding Strategy On the network, each cluster has a unique ID, representing the location of the cluster in the whole semantic space and being the basis of establishing neighborhood. This paper offers a cluster encoding algorithm, which automatically generates an ID for a new cluster. The system uses a binary number of m bits as a cluster ID, and each cluster maintains a variable Par_times to record the partitioning time (used for generating a new ID). The first cluster ID is predefined as 0 and Par_times as 0. When the cluster is partitioned with peers joining, the newly generated cluster m-Par_times-1 obtains a new ID that equals the ID of the original cluster plus 2 , and Par_times is increased by one. When Par_times is more than m, partitioning can no longer be done, and therefore m should be large enough. The cluster encoding algorithm is Algorithm 1. Algorithm 1. Algorithm for cluster encoding Input: IDold: ID of the original cluster; Par_times : partitioning times; m: length of a cluster ID Output: IDnew: ID of the new cluster 01: if Par_times<m then 02: IDnew = IDold + 2m-Par_times-1 03: Par_times= Par_times+1 04: End if

For example, we assume that the length of cluster codes m = 4, i.e. binary numbers have 4 bits. At the beginning there is one cluster in the system shown in Fig. 3, whose semantic space is the whole taxonomy tree. According to the cluster encoding strategy, its cluster ID is 0, denoted by SC0, Par_times = 0, the ID of the new cluster generated by the first time partitioning is 0 + 24-1 = 8, denoted by SC8 shown in Fig. 4, and Par_times changes into 1, kept in the two clusters prepared for the next partitioning.

A TBA for Constructing Semantics-Based Super-Peer Networks

127

3.4 Neighborhood Establishment Among Clusters Neighborhood establishment among clusters exerts significant effluence on P2P network performance and is directly related to query efficiency and network scalability. On a super-peer network, super-peers are responsible for maintaining and dealing with routing among clusters. In the system, each super-peer maintains a cluster information table that stores relevant status and routing information. The table is shown in Table 1, there are five fields: Cluster_ID is the cluster ID, Par_times is the partitioning time, Cluster_size is the load size of the cluster, Cluster_range is current semantic sub-space of the cluster, represented by a set of IDs of root nodes of taxonomy sub-trees, and Network_range is initially semantic sub-space of the cluster when the cluster is generated, represented by a set of IDs of root nodes of taxonomy sub-trees, representing semantic sub-spaces of the cluster and those partitioned apart from it. Obviously Cluster_range is a subset of Network_range. Neighbor_list is a list of neighbor clusters, comprised of Network_range’s and Cluster_ID’s of neighbor clusters. In the system, routing information among super-peers is dynamically built when partitioning clusters according to cluster IDs. If there is one and only one bit difference between two cluster IDs, the two clusters are neighbors. A cluster and a new cluster partitioned from it must be neighbors according to cluster encoding, it can effectively reduce routing hops when searching, which decreases communicating cost of the system and enhances search performance as a result. Table 1. Information table of a cluster Cluster_id Cluster_range Par_times

Network_range Cluster_size Neighbor_list: { Network_range, Cluster_id}

The process of adding neighbors is: when a cluster is partitioned, the new cluster and the original one are neighbors, corresponding information is added to their cluster information tables, and the new cluster broadcasts its ID to network, and finds clusters whose IDs has one and only one bit difference from its ID. If such clusters are found within a given hops, corresponding information is added to the respective cluster information tables and thus neighborhood is established. For example, as shown in Fig. 4, clusters SC0 and SC8 continue being partitioned with load increase, SC12 is partitioned from SC8, SC4 from SC0, SC6 from SC4, and finally obtains the result (Fig. 5). After several times of partitioning, the original cluster is partitioned into five clusters. Accordingly a taxonomy tree is partitioned into several taxonomy sub-trees. According to the mechanism of neighborhood establishment, the neighborhood among the five clusters and their cluster information are shown in Fig. 6. For instance, the cluster SC8, whose Cluster_ID is 10002, Par_times is 2, and Cluster_size is 52. At the time the cluster is generated, the cluster consists of two taxonomy sub-trees whose root IDs are 0.2 and 0.3, and thus its Network_range is the set {0.2, 0.3}, and now the cluster is comprised of the sub-tree whose root ID is 0.2, so its Cluster_range is {0.2}. Since SC0 generates SC8 and SC8 generates SC12, SC8 has two neighbor clusters SC12 and SC0. Cluster information of other clusters is shown in Fig. 6.

128

B. Qiao, G. Wang, and K. Xie

SC8 (1000)

SC12 (1100)

0.2 12

0.3 9

0 12

20 19 16 0.2.1 0.2.2 0.2.3

17 16 11 0.3.1 0.3.2 0.3.3

SC4 (0100)

SC4 (0100)

42 13 65 0.1.1 0.1.2 0.1.3

42 0.1.1

SC0 (0000)

45 0.1 SC6 (0110) 65 0.1.3

13 0.1.2

Fig. 5. Results after re-clustering based on the results in Fig. 4

Cluster_id =4, Par_times=3 Cluster_size =55 Network_range ={0.1.1,0.1.2,0.1.3} Cluster_range={0.1.1,0.1.2} Neighbor list= {(0,{0}),(6,{0.1.3}) , (12,{0.3}) }

Cluster_id=0; Par_times=2 Cluster_size=57 Network_range {0} Cluster_range={0} Neighbor list= {(8,{0.2,0.3}), 4,{0.1.1,0.1.2,0.1.3})}

SC0

Cluster_Id=8, Par_times=2 Cluster_size =67 Network_range {0.2,0.3} Cluster_range={0.2} Neighbor list= {(0,{0}),(12,{0.3}) } SC8

SC4

Cluster_id =6, Par_times=3 Cluster_size =65 Network_range {0.1.3} Cluster_range={0.1.3} Neighbor list= { (4,{ 0.1.1,0.1.2,0.1.3}) }

Q1: 0.2.2

SC12

SC6

Cluster_id =12, Par_times=2 Cluster_size =53 Network_range {0.3} Cluster_range={0.3} Neighbor list= {(8,{0.2,0.3}), (4,{0.1.1,0.1.2,0.1.3}) }

Fig. 6. Neighborhood of the five clusters

4 Query and Network Maintenance 4.1 Query-Routing Algorithm Since semantics of data and query is described with taxonomy hierarchy, each query is associated with a concept of the taxonomy hierarchy, which is corresponding to a node on the taxonomy tree, the node express semantics of the query; we name ID of the node as Search ID (SID). According to structural characteristics of the taxonomy tree, when a node is the searching target of a query, its ascendants and descendants are searching targets as well, and hence the query is sent to the clusters that contain the target nodes. In this paper, the ascendant-descendant relationship is judged by node IDs of the taxonomy tree. The query-routing process is as follows. When a peer in the cluster A sends a query Q to its connected super-peer, the super-peer obtains the SID from Q’s semantic information. If SID has ascendantdescendant relationship with some ID(s) in A’s Cluster_range, Q is executed by finding all peers in A’s index that can answer Q, sending Q to these peers to execute and returning results to the peer that initially sends Q. At the same time, it is checked

A TBA for Constructing Semantics-Based Super-Peer Networks

129

whether in each Network_range in A’s Neighbor_list, there is some ID of ascendantdescendant relationship with SID. If so, Q is sent to these neighbor clusters. If in A’s Cluster_range and any Network_range in the Neighbor_list, there is no node whose ID has ascendant-descendant relationship with SID, Q is forward to the neighbor cluster B that generates A, and then B continues to process Q. Algorithm 2 shows how to query and route. As shown in Fig. 6, a peer in the cluster SC6 sends a query Q1 whose SID is 0.2.2. According to Algorithm 2, the routing path is shown in the form of arrows in Fig. 6. Since no ID in SC6’s Cluster_range and any of Network_range in its Neighbor_list have ascendant-descendant relationship with the SID, Q1 is routed to SC4, which generates SC6. The ID 0 in Network_range in SC4’s Neighbor_list has ascendantdescendant relationship with the SID, and then Q1 is forward to the neighbor SC0. According to its Cluster_range, SC0 is recognized as one of destinations of Q1, and meantime Q1 is forwarded to SC8 according to the Network_range’s in SC0’s Neighbor_list, and thus SC8 is another destination of Q1. Algorithm 2. Query-routing algorithm Input: Cid: cluster ID, SID 1: If received Search(SID) before then 2: Drop Search(SID) 3: Return; 4: end if 5: Search_token=0; 6: If exist a ascendant-descendant relationship between SID and a ID in cluster_range then 7: local_index_search(SID); 8: Search_token=1; 9: End if 10: For any Network_range NR in neighbor_list do

；

11:

12: 13: 14: 15: 16: 17: 18: 19:

20:

If exist a ascendant-descendant relationship between SID and a ID in NR then Forward Search(SID) to neighbor Cluster_id Search_token=1; End if End for If Search_token!=1 then X=Cid XOR (2m-1); J=[x OR (x+1)] XOR (2m-1); Forward Search(SID) to neighbor cluster J; End if

4.2 Network Maintenance 1. Peer join. When a peer P joins, he should find an existing cluster C in the network firstly, and send semantic index of its Data to C, C routes the semantic index in the overlays, and finally the semantic index is stored on the cluster B according to its semantics. B is responsible for saving connection and other relevant information of P, and meantime B joins P. This is how peer joins 2. Peer leave. When a peer P leaves, P sends a leaving request to its connected superpeer and then it leaves. The super-peer deletes relevant information and modifies Cluster_size value in the cluster information table. 3. Peer failure. The super-peer of a cluster periodically checks availability of peers it manages. If a peer fails, it is labeled with temporary failure status in the corresponding index, and if temporary failure status remains over a certain period of time, the peer is regarded to leave and the same thing is done as peer leave.

130

B. Qiao, G. Wang, and K. Xie

5 Performance Evaluation 5.1 Experimental Environment Setup In order to verify validity of TBA, we evaluate the performance using simulating experiment. The experimental environment is P4 3.0GHz, 512M memory, and 80G hard disk, the operating system is Windows server 2003 and JDK 5.0 as the programming environment to develop the stimulating test software. In the experiment synthesized data are employed to test performance of the algorithm. There are many taxonomy trees of 2-10 layers composed of 200-100K concepts (nodes) of 2-30 fanouts. For calculation convenience, it is assumed that each peer relates to only one index item, and thus the load size of a cluster can be viewed as the number of peers. For universality, distribution of peers on the taxonomy tree follows the Zipf-like (α=0.8) distribution. At the beginning there is one cluster and one super-peer in the system. With peers continuously joining, the cluster is automatically partitioned when the maximal cluster load size M is exceeded. 5.2 Experimental Results In the paper, experimental research focuses on three aspects: network scalability, average similarity of data within a cluster and load skew degree. Comparison with a representative constructing approach, the DHT-based approach [15] is made. Experimental results are detailed as follows.

TBA

35

DHT-based

30 25 20 15 10 5 0 256

512

1024

2048

4096

8192

16384

Network size (number of peers)

Average message number per query(AMNPQ)

Average query hops(AQH)

(1) Network scalability Network scalability mainly evaluates how much query efficiency and cost change when the network size varies. In this paper the average query hops (AQH) is used to evaluate query efficiency and average message number per query (AMNPQ) to evaluate query cost. We construct a 4-layered taxonomy tree composed of 2808 concepts. When the maximal cluster load size M is 100 and 10 queries are sent randomly from each peer, the AQH with network size varying are shown in Fig. 7. It can be seen AQH increases slowly with network size increasing, and the AQH of the TBA is lower than that of DHT-based method. For TBA clusters peers based on their data semantics and routes semantically, and thus reduces the communication times inter clusters, the AQH consequently decreases. The DHT-based method can not TBA

80

DHT-based

70 60 50 40 30 20 10 0 256

512

1024

2048

4096

8192

16384

Network size (number of peers)

Fig. 7. Comparison of AQH with the network Fig. 8. Comparison of AMNPQ with the size varying based on two different approaches network size varying based on two different approaches

A TBA for Constructing Semantics-Based Super-Peer Networks

131

support semantic clustering and thus data semantics in a cluster is loosen, so the AQH is higher. Fig. 8 shows how the AMNPQ change in the same condition as Fig. 7. It can be seen that the AMNPQ slowly increase with peers increasing almost in consistent with changing of AQH. From the two aspects, it can be seen that TBA is better. (2) Average similarity of data within a cluster In this paper average similarity of data within a cluster is used to evaluate semantic similarity of data in a cluster after partitioning. Average similarity of data is denoted by Sim, shown in Formula (1) Sim =

1 n ∑1/Di n i=1

(1)

Where n is the number of clusters on the network, Di is the average distance of roots of all taxonomy sub-trees in the i-th cluster. The distance of two nodes on a taxonomy tree equals the number of the edges from the two nodes to their first common ascendant node. It is obvious that the larger Di is, the larger the distance of the nodes is, the less the similarity is, and the smaller Sim is. On the other hand, the smaller Di is, the less the distance of the nodes is, the more the similarity is, and the larger Sim is. For a cluster composed of one taxonomy sub-tress, since there is only one root, we predefine Di is 1. Fig.9 shows comparison of average similarities of data within a cluster generated in two approaches in the same condition as Fig.7. It can be seen that the average similarity of data within clusters by TBA is higher than that by the DHT-based approach. With the network size varying, the two average similarities grow larger and closer to each other, both approximating to 1. This indicates that TBA can well maintain semantic relationship of data within a cluster and the average similarity grows more with the network size increasing. (3) Load skew degree Load skew degree represents how load skews among clusters, calculated by the standard deviation of loads among clusters, denoted by LSD in Formula (2) LSD =

n

∑ (x

i

− x )2 / n

(2)

1

Where xi is the load of the i-th cluster and x is the average load of all clusters. Obviously the larger LSD is, the larger the difference of cluster loads is, indicating the less balance clusters have, whereas the less LSD is, the less the difference of cluster loads is, indicating the more balance clusters have. Fig.10 shows comparison of Load skew degrees of clusters by the two approaches in the same condition as Fig. 7. Load skew degrees by the two approaches fluctuate regularly and both relatively slightly, and with network size increasing, the two LSDs fluctuate gradually less and approach each other. In view of load skew degree, both approaches maintain load-balancing among clusters, TBA is better than DHT-based approach, but the advantage of TBA over DHT-based approach becomes less and less with network size increasing.

B. Qiao, G. Wang, and K. Xie

TBA

Average similarity of data within a cluster

1

DHT-based

0.8 0.6 0.4 0.2 0 128

256

512

1024

2048

4096

8192

Network size (number of peers)

16384

load skew degree (LSD)

132

TBA

45 40 35 30 25 20 15 10 5 0 128

256

512

DHT-based

1024

2048

4096

8192

16384

Network size (number of peers)

Fig. 9. Comparison of average similarity of Fig. 10. Comparison of load skew degrees data within a cluster based on two approaches (LSD) based on two approaches

6 Related Works Much research has been focused on improving search efficiency and scalability in P2P systems. Several techniques have been proposed to improve search in unstructured P2P systems, such as constructing routing indices of information of related neighbors on each peer [8,9], using of cache [10,11], using histograms to direct queries [2,18], etc. In structured P2P systems, most of research constructs a system based on distributed hash tables, such as CHORD[3], PASTY[4], CAN[5] , etc. Another effective method to improve query performance is partitioning the network into distinct semantic clusters and constructing SONs (Semantic Overlay Networks), each of which is a semantic cluster. Queries are merely forwarded to the SON that probably contains answers. The advantage of this approach is that it reduces the search cost and enhances search efficiency and scalability of the system. The concept of SONs is firstly proposed in paper [12], where SONs are constructed by taxonomy hierarchy, peers are distributed on a concept in taxonomy hierarchy and the concept is regarded as a SON, however, without providing how to manage these SONs; paper [13] proposes a dynamic clustering method in unstructured systems based on multidimensional semantic space, not suitable for taxonomy hierarchy semantic space; paper [14] proposes a method of creating SONs in structured P2P networks, which uses LSI to produce semantic vectors of documents, which are stored in an CAN, and uses nearest neighbor query technology; paper [15] proposes an education resource sharing model based on super-peer, in which peers are classified by taxonomy hierarchy, and taxonomy hierarchy is distributed on super-peers by hash functions, which results in loose semantics of peers in one cluster and hence increases search cost; paper [16] proposes a routing and clustering strategy based on routing indices in super-peer networks, in which concepts and schemas are used to cluster peers without considering load-balancing among super-peers; paper [17] proposes a clustering architecture based on super-peer networks, in which the semantic schema of every cluster is predefined and peers are distributed into several fixed clusters according to the match between peers and clusters.

A TBA for Constructing Semantics-Based Super-Peer Networks

133

7 Conclusions In this paper, we focus on how to construct a semantics-based P2P network using semantic information contained by a domain taxonomy hierarchy, and propose a selforganized approach based on the taxonomy tree. We also give relevant algorithms and make comparison with the DHT-based approach. Experiments show that our approach has low search latencies and overheads, and load-balancing among super-peers is well maintained. Future work focuses on the dynamical load-balancing algorithms among clusters and maintenance approaches in the case of super-peer leave and failure based on improving clustering and the query-routing algorithm. Acknowledgments. This research was supported by National Basic Research Program of China (2006CB303103), the National Natural Science Foundation of China (Grant No. 60473074 and 60573089), Cultivation Fund of the Key Scientific and Technical Innovation Project (Grant No. 706016) and Natural Science Foundation of Liaoning Province (Grant No. 20052031).

References 1. Gnutelliums, LLC.: Gnutella Protocol Specification Version 0.4, 2004-02-18 (2004), http://www.limewire.com/developer/gnutella-protocol-0.4.pdf 2. Löser, A., Tempich, C.: On Ranking Peers in Semantic Overlay Networks. In: Althoff, K.D., Dengel, A., Bergmann, R., Nick, M., Roth-Berghofer, T.R. (eds.) WM 2005. LNCS (LNAI), vol. 3782, Springer, Heidelberg (April 2005) 3. Stoica, I., Morris, R., Karger, D., Kaashoek, M.F.: Chord: A scalable peer-to-peer lookup service for internet applications. In: Proc. ACM SIGCOMM (August 2001) 4. Rowstron, A., Druschel, P.: Pastry: Scalable, distributed object location and routing for large-scale peer-to-peer systems. In: Proc. of the 18th IFIP/ACM International Conference on Distributed Systems Platforms (Middleware 2001) (November 2001) 5. Ratnasamy, S., Francis, P., Handley, M., Karp, R., Shenker, S.: A scalable contentaddressable network. In: Proc. ACM SIGCOMM (August 2001) 6. Nejdl, W., Wolf, B., Qu, C., et al.: EDUTELLA: a P2P Networking Infrastructure based on RDF. In: WWW 11 Conference Proceedings, Hawaii, USA, (May 2002) 7. KaZaA., http://www.kazaa.com 8. Pireddu, L., Nascimento, M.: Taxonomy-based routing indices for peer-to-peer networks. In: Proceedings of the SIGIR Workshop on P2P IR (2004) 9. Tempich, C., Staab, S., Wranik, A.: REMINDIN’: Semantic Query Routing in Peer-toPeer Networks based on Social Metaphors. In: Proceedings of the 13th International Conference on World Wide Web (WWW’2004) (2004) 10. Bhattacharjee, et al.: Efficient Peer-to-Peer Searches Using Result-caching. In: Proceeding of the 2nd IPTPS. (2003) 11. Wang, C., Xiao, L., Liu, Y.: Distributed Caching and Adaptive Search in Multilayer P2P Networks. In: Proceedings of ICDCS’04 (2004) 12. Crespo, A., Garcia-Molina, H.: Semantic Overlay Networks for P2P Systems. Technical report, Stanford University (2002)

134

B. Qiao, G. Wang, and K. Xie

13. Li, M., Lee, W.C., Sivasubramaniam, A.: Semantic Small World: An Overlay Network for Peer-to-Peer Search. In: Proceedings of the International Conference on Network Protocols (October 2004) 14. Tang, C., Xu, Z., Dwarkadas, S.: Peer-to-Peer Information Retrieval Using SelfOrganizing Semantic Overlay Networks. In: Proceedings of SIGCOMM’03 (2003) 15. Loeser, A.: Taxonomy-based Overlay Networks for P2P Systems. In: Proceedings of IDEAS’04 (2004) 16. Nejdl, et al.: Super-Peer-based Routing and Clustering Strategies for RDF-based P2P Networks. In: Proceedings of WWW’03 (2003) 17. Eisenhardt, M., Mueller, W., Henrich, A.: Classifying Documents by Distributed P2P Clustering. GI Jahrestagung, pp. 286–291 (2003) 18. Petrakis, Y., Koloniari, G., Pitoura, E.: On Using Histograms as Routing Indexes in Peerto-Peer Systems. In: DBISP2P (2004) 19. Qiao, B., Wang, G., Xie, K.: A self-organized Semantic Clustering Approach for Superpeer Networks. In: WISE2006 (October 2006)

A Comparative Study of Replica Placement Strategies in Data Grids Qaisar Rasool1, Jianzhong Li1,2, George S. Oreku1, Ehsan Ullah Munir1, and Donghua Yang1 1

School of Computer Science and Technology, Harbin Institute of Technology, China 2 School of Computer Science and Technology, Heilongjiang University, China [email protected], [email protected], [email protected], [email protected], [email protected]

Abstract. Data Grids are today’s emerging infrastructure providing specialized services on handling large datasets that needs to be transferred and replicated among different grid sites. Data replication is an important technique for data availability and fast access. In this paper we present a comparison of various replication models and techniques employed by some major topologies used in data grid environment. We focus on dynamic strategies for replica placement in tree, Peer-to-Peer (P2P) and hybrid architectures. Beside tree model which is being implemented for many Data Grid applications, hybrid and P2P grid models of replication are also emerging for providing scientific communities with better availability and efficient access to massive data.

1 Introduction Grid computing is a wide-area distributed computing environment that involves largescale resource sharing among collaborations, referred to as Virtual Organizations, of individuals or institutes located in geographically dispersed areas. A Data Grid [1] provides services that help users discover, transfer and manipulate large datasets stored in distributed repositories and also, create and manage copies of these datasets. Examples of the scientific applications dealing with huge amounts of data and the potential beneficiaries of Data Grid technology are high energy physics, astronomy, bioinformatics, and earth sciences. Replication of data means the creation and maintenance of copies of data at multiple computers. Replication is a key to the effectiveness of distributed systems in that it can provide enhanced performance, high availability and fault tolerance. Grid being a specialized form of distributed system is going to get benefit from replication and possibilities are continually explored by the research community. In this paper we present a brief review of current and past research on replication techniques. Specifically, we focus on data replica placement policies proposed for use in the data grid environment. For each replica placement technique, we consider its methodology, objective (or performance metric), and results. In the presence of diverse and varying characteristics of tree and other architectures it is difficult to create a common ground for comparison of different replication strategies. We, K.C. Chang et al. (Eds.): APWeb/WAIM 2007 Ws, LNCS 4537, pp. 135–143, 2007. © Springer-Verlag Berlin Heidelberg 2007

136

Q. Rasool et al.

therefore, group topologies into tree and hybrid/P2P architectures and analyze the impact of replica placement policies in each one. A hybrid topology can carry features of both tree and P2P architectures and thus can be used for better performance of a replication strategy. The paper is organized as follows. Section 2 describes some widely used grid topologies and their implications. In section 3, we take a closer look on data replication in grid environment and how different replica placement strategies work under analogous topologies. Section 4 concludes the paper.

2 Grid Topologies In this section we present an overview of major grid topologies. We use terms tree, hierarchical, and multi-tier to refer the same topology. Other topologies considered are P2P and hybrid. A hierarchical or tree model is used where there is a single source for data and the data has to be distributed among collaborations worldwide. For instance, the data grid envisioned by the GriPhyN project is hierarchical in nature and is organized in tiers [2]. The source where the data is produced is denoted as Tier 0 (e.g. CERN). Next are the Tier 1 national centers, the Tier 2 regional centers (RC), the Tier 3 workgroups and finally Tier 4, which consists of thousands of desktops. Such a multi-tier Data Grid has many advantages [3]. Firstly, it allows the scientific community to access the resources in a common and efficient way. Secondly, the datasets can be distributed to appropriate resources and accessed by multiple sites. The network bandwidth can be used efficiently because most of the data transfers involve local or national network resources, hence alleviating the workload of international network links. Thirdly, with the support of the Grid middlewares, the resources located in different centers can be utilized to support data-intensive computing. The multi-tier Data Grid model is being implemented for many scientific applications, the High Energy Physics (HEP) being the most prominent one.

(a)

(b)

(c)

Fig. 1. Data grid topologies (a) tree, (b) P2P, and (c) hybrid

A tree topology also has shortcomings. The tree structure of the grid means that there are specific paths the messages and files can travel to get to the destination. Further, data transference is not possible among sibling nodes or nodes situated on the same tier. Peer-to-Peer (P2P) systems overcome these limitations and offer flexibility in communication among components. A P2P system is characterized by

A Comparative Study of Replica Placement Strategies in Data Grids

137

the applications that employ distributed resources to perform functions in a decentralized manner. From the viewpoint of resource sharing, a P2P system overlaps a grid system. The key characteristics that distinguishes a P2P system from other resource sharing systems is its symmetric communication model between peers, each of which acts as both a server and a client. The unique characteristics of P2P make the Data Grid more scalable and flexible to deal with the distributed data and replication. With the needs for researchers to collaborate and share products of their analysis, new hybrid models are emerging for Data Grids which utilize the combined benefits of tree and other architectures such as ring and P2P. A hybrid model of a hierarchical Data Grid with peer linkages at the edges is shown in Fig.1(c).

3 Data Replication in Grid Replication is used in distributed systems to improve data access efficiency and fault tolerance. Fig.2 shows pros and cons of data replication. The general idea is to store copies of data in different locations so that data can be easily recovered if one copy of data is lost. Also, placing data replicas closer to user can improve data access performance significantly. A replica is an exact copy of a file that is linked to the original file through some well-defined mechanisms [4]. In Grid environment, replica management services do not enforce any semantics to (multiple) replicas of a file and thus a replica takes the form of a user-asserted correspondence between two physical files [1]. Data replication is characterized as an important optimization technique in Grid for promoting high data availability, low bandwidth consumption, increased fault tolerance, and improved scalability.

Fig. 2. Pros and cons of data replication in a generic distributed system

The goals of replica optimization is to minimize file access times by pointing access requests to appropriate replicas and pro-actively replicating frequently used files based on access statistics gathered. In general, replication mechanism determines which files should be replicated, when to create new replicas, and where the new replicas should be placed. Replication schemes can be classified as static and dynamic. In static replication, a replica persists until it is deleted by users or its duration is expired. The drawback of static replication is evident, when client access patterns change greatly in the Data

138

Q. Rasool et al.

Grid, the benefits brought by replica will decrease sharply. In contrast to static replication, dynamic replication automatically creates and deletes replicas according to changing access patterns, and thus ensures that the benefits of replication continue even if user behavior changes. Dynamic replication strategies tend to reduce hotspots created by popular data and facilitate load sharing. The benefits achieved, however, incur extra overhead costs in the form of real-time assessment of network traffic and maintaining efficient system logs to handle the dynamicity of grid environment. 3.1 Replica Placement in Tree Topology Grids The manner in which replicas are placed in the grid nodes follows two approaches. The first approach is to distribute data from top to bottom starting from root node to lower tier nodes. The other technique is to place replicas in bottom up fashion. There is a variation in the consideration of read/write costs before making replication decision. We take both approaches into consideration in our discussion. As most of the data in Data Grids are read-only [5, 6], we ignore replica update and consistency issue. We start with the initial work [7] on replication strategies that are proposed and simulated for a hierarchical Data Grid similar to Fig.1(a). These techniques are briefly described as follows: 1) No Replication or caching: all data held at root (the base case against which other strategies are evaluated); 2) Best Client: The access history of files is maintained and a replica of a file is created at the client node who has generated most number of requests for that file; 3) Cascading Replication: when file access exceeds a certain threshold, a replica of file is created at immediate lower tier node on the path to the best client; 4) Plain Caching: the client that requests a file stores a copy locally; 5) Fast Spread: When a client requests a file, a copy (replica) is stored at each node on the path to the client. Another policy named Cascading plus Caching was also proposed in which a client node can act as server for its siblings. We discuss this strategy in the next section. It is important to note the difference between replication and caching. In case of replication, the decision to make a replica and sending it to other node is solely taken by the server. In case of caching, a client requests a file and server sends a copy of the file to be stored in the local cache of client. We can say that caching is a client-side phenomenon whereas replication is a server-side phenomenon. The above algorithms consider three different patterns of data access: 1) Random (no locality in data); 2) Access pattern with assumption that popular files at one site are popular at others (temporal locality); 3) Access pattern with little temporal locality and assumption that files recently accessed by a client are likely to be accessed by nearby clients (geographical locality). In order to capture the dynamics of the access pattern, the history statistics (popularity logs) are cleared at specified time intervals. The effectiveness of a replication strategy is related to how well the time interval is tuned to the access behavior (i.e. interval checking); another parameter is the threshold (only if number of requests exceed this threshold is the file replicated). Among these strategies, Fast Spread shows a relatively consistent performance and best in case of random access pattern. When locality is introduced, Cascading offers good results. In contrast, Best Client presents the worst case among all. For locating

A Comparative Study of Replica Placement Strategies in Data Grids

139

the nearest replica each model/algorithm selects the replica server site that is at the least number of hops from the requesting client. An improvement in Cascading technique is proposed namely Proportional Share Replica policy [8]. The method is a heuristic one that places replicas on the optimal locations by assuming that number of sites and the total replicas to be distributed are already known. Firstly an ideal load distribution is calculated and then replicas are placed on candidate sites that can service replica requests slightly greater than or equal to that ideal load. The formula for calculating ideal load is:

Load = Totalrequests /(Originalcopy + Replicas ) Table 1. Merits and demerits of replica placement techniques in tree topology Grids Authors

Technique

Best client

Cascading Kavitha and Foster [7] 2001

A replica created at node who generates max requests Replica trickles down to lower tier if no. of requests exceeds threshold

Results/Comments

TD /B U*

Objective/ Performance metric

TD

Faster avg response time and bandwidth conservation

TD

Mean response time

- worst performance - not suitable for grid + better for small degree of locality - Bad for random access pattern + almost similar performance as cascading - relatively high response time + consistent performance + best for random access pattern - high storage req, I/O, CPU load + better results over Cascading + load sharing among replica servers

Caching

A requesting client receives and stores a copy locally

Fast Spread

A replica is stored at each node along the path toward client

Abawajy [8] 2004

Proportional Share Replication

Calculating an ideal workload and distributing replicas

Ming Tang et al. [3] 2005

Dynamic Replication Algorithms

Simple Bottom Up (SBU), Aggregate Bottom Up (ABU)

+ better results over Fast Spread strategy

B U

Response time, bandwidth cost, replication freq

Multiobjective approach

Solved as pfacility problem to get sites for replica placement

+ user requests and currentnetwork status + dynamic maintainability when performance metric degrades

TD

Requestweighted average response time

Rahman et al. [9,10] 2005, 2006 *

Methodology/ Realization

TD = Top-down, BU = Bottom-up.

An intensive work in the form of dynamic replication algorithms is presented by Tang et al [3]. Nodes at the intermediate tiers (i.e. tiers between root and client nodes)

140

Q. Rasool et al.

act as the replica servers, each running a Local Replica Manager (LRM). A Dynamic Replication Scheduler (DRS) makes the replication decision after gathering file access statistics. The DRS uses two algorithms, Simple Bottom-Up (SBU) and Aggregate Bottom-Up (ABU). With SBU, replicas are created as close as possible to the clients with requests rates exceeding the predefined threshold. The algorithm ABU takes into account the access history of files used by the sibling nodes and aggregates the access record of similar files so that these frequently accessed files are replicated first. The performance of algorithms was evaluated and improvements shown against Fast Spread dynamic replication strategy. The values for interval checking and threshold were based on data access arrival rate, data access distribution and capacity of the replica servers. A method exploiting Operations Research techniques is proposed in [9,10] for replica placement. In this method, named multi-objective approach, replica placement decision is made considering both the current network status and data request pattern. The problem is formulated in p-median and p-center models to find the p replica placement sites. The p-center problem targets to minimize the max response time between user site and replica server whereas the p-median model focuses on minimizing the total response time between the requesting sites and the replication sites. The dynamic maintainability is achieved by considering the replica relocation cost. The decision of relocation is made when performance metric degrades significantly in last K time periods. The threshold value is varied proportionally to response time in each time interval. Table 1 summarizes the major research work done on replica placement in tree topology Data Grids. 3.2 Peer-to-Peer and Hybrid Solutions for Replica Placement In large decentralized systems where the reliability of storage elements is low and bandwidth is limited, data availability is a serious challenge. Replication can be an effective technique for improving data availability in such unreliable systems. A strategy for creating replicas automatically in a generic decentralized peer-to-peer network is proposed in [11]. The goal of this model-driven approach is to maintain a specific level of availability at all times. To achieve this goal, the optimal number of replicas for each file is calculated and then best nodes are identified to host these replicas. This is done for files that have no replica or have replicas less than an availability threshold. The approach relies on a resource discovery service to find available storage and the Network Weather Service [14] to get network status information. The availability of a file depends on the node failure rate and the accuracy of the Replica Location Service (RLS). The amount of replica availability needed for a given file is modeled as:

RLacc *(1 − (1 − p) r ) ≥ Avail Where r is the total number of replicas for a given file, p is the probability of a node to be up, RLacc is the accuracy of the replica location service, and Avail is the required amount of availability for the file. For a certain availability threshold, the value of r can be calculated from the above function. For example, if the probability of a node to be up is 30% and the RLS accuracy is 80% then for the availability threshold of 75%,

A Comparative Study of Replica Placement Strategies in Data Grids

141

the model recommends a minimum of 8 replicas. After calculating ideal number r of replicas, the system consults RLS to know how many replicas M are actually present. If M is less than r the system creates (r – M) copies of file and distribute them to best candidate nodes. The best nodes are those which maximize the difference replicationBenefits – replicationCosts. The benefit is the reduction in transfer time to the potential users. The replication costs are the storage costs at the remote site and the transfer time from the current location to the new location. Let F be a file stored at a location N1 then cost of creating its replica at a new location N2 is:

replicationCosts = trans ( F , N 1, N 2) + s ( F , N 2) where function trans() shows the transfer cost of F from N1 to N2 and function s() represents the storage cost of F at N2. The benefit of creating a replica of F at N2 is given by:

replicationBenefits = trans ( F , N 1,User ) − trans ( F , N 2, User ) where User is the location from which we expect the most number of future requests. In this way, the net benefits of replicating F at N2 are calculated and the best candidate identified to place replica. Table 2. Merits and demerits of replica placement techniques in P2P & hybrid topology Grids Authors

Kavitha et al [11] 2002 Hai Jin et al [12] 2006 Kavitha and Foster [7] 2001 Lameha -medi etal [13] 2002

Objective/ Performance metric

Technique

Methodology/ Realization

Model-driven P2P Replication

Computing ideal no. of replicas for a file; placing replicas at best nodes by doing cost/benefit analysis

Decentralized P2P Strategy

Modeling costs b/w peers using adjacency graph; 0/1 vector for replica placement

+ better performance than static technique

Data transfer cost

Cascading plus Caching

Joining the cascading and caching methodologies

+ Client can act as Server for siblings + better performance than cascading or caching alone

Response time and bandwidth conservation

Hybrid Replica Connection Service

A cost model evaluates access costs and performance gains of creating & placing replicas

+ Favorable results when replicas placed closer to users

Response time

Results/Comments

+ No single point of failure - possibility of excessive replicas as each peer takes own decision

Replica availability

In [12], a P2P network cost model is constructed along with a decentralized algorithm to dynamically optimize the replica placement. The optimization strategy is based on the status of the grid system. Though all the status info is obtained dynamically in a decentralized way and all decision-making of replica movement is independent among the peer nodes, the data locating and storage resource

142

Q. Rasool et al.

management are centralized. System scale is controlled by deploying super peers or brokers. To make the effective use of grid storage resources, the lifetimes of replicas are set. A replica will be deleted from the system if it is not accessed for a certain time period. Among replication techniques for tree topology discussed in previous section, authors [7] also presented Caching plus Cascading replication strategy by joining Plain Caching and Cascading. A copy of the requested file is saved at the client local storage and periodically replicas of popular files are sent down to lower tiers. Any node in the hierarchy can be server. Specifically, a client can act as a Server to its siblings. This constitutes a hybrid topology in which client nodes having the same parent are linked in P2P-like structure. The strategy gives better performance than Cascading or Caching alone. A hybrid topology is used in [13] where ring and fat-tree replica organizations are combined into multi-level hierarchies. Each site or node maintains an index of the replicas it hosts and the other locations of these replicas that it knows. Replication of a dataset is triggered when requests for it at a site exceed some threshold. A cost model evaluates the data access costs and performance gains for creating replicas. The replication strategy places a replica at a site that minimizes the total access costs including both read and write costs for the datasets. Table 2 highlights some major work on replica placement in P2P and hybrid Data Grid topology.

4 Conclusions Comparing different replication policies working under different or even similar topologies is complicated due to diverse nature of assumptions made regarding topology, data access patterns and policy objectives. The hierarchical model of Data Grid, used in many high energy physics projects, has motivated the development of tree-structured replication mechanisms. The decision of replica placement is very important for the effective performance of any replication scheme. For our study, we consider tree and P2P/hybrid structures separately and present the merits and demerits of different replica placement strategies in each one. A Data Grid tree provides a relatively simple top-down structure for data propagation, albeit data movement remains confined to specific paths. In contrast, Peer-to-Peer systems present a more flexible layout but at the cost of additive complexity in the replication schemes. There are proposals for P2P and hybrid replication solutions in research but practically they have not yet found a place in Data Grids. We foresee that like tree structure, P2P and other replication frameworks shall become prevalent to fulfill the needs of scientific communities for accessing huge amounts of data efficiently and securely.

Acknowledgements We would like to thank the National Natural Science Foundation of China under Grant No. 60533110 and 60473075, the Program for New Century Excellent Talents

A Comparative Study of Replica Placement Strategies in Data Grids

143

in University under Grant No.NCEF-05-0333, the Key National Science Foundation of Heilongjiang Province under Grant zjg03-05 and the National Science Foundation of Heilongjiang Province under Grant F0208 for their support.

References 1. Chervenak, A., Foster, I., Kesselman, C., Salisbury, C., Tuecke, S.: The Data Grid: Towards an Architecture for the Distributed Management and Analysis of Large Scientific Datasets. In. Journal of Network and Computer Applications (23), 187–200 (2001) 2. The GriPhyN Project, http://www.griphyn.org 3. Tang, M., Lee, B.-S., Yeo, C.-K., Tang, X.: Dynamic Replication Algorithms for the Multi-tier Data Grid. In Future Generation Computer Systems 21, 775–790 (2005) 4. Guy, L., Kunszt, P., Laure, E., Stockinger, H., Stockinger, K.: Replica Management in Data Grids. Technical Report, Global Grid Forum Information Document, GGF5, Edinburgh, Scotland (July 2002) 5. Stockinger, H., Samar, A., Allcock, B., Foster, I., Holtman, K., Tierney, B.: File and Object Replication in Data Grids. In: 10th IEEE Symposium on High Performance and Distributed Computing. pp. 305–314 (2001) 6. Hoschek, W., Janez, F.J., Samar, A., Stockinger, H., Stockinger, K.: Data Management in an International Data Grid Project. In: proceedings of GRID Workshop. pp. 75–86 (2001) 7. Ranganathan, K., Foster, I.: Identifying Dynamic Replication Strategies for a High Performance Data Grid. In: proceedings of the Int’l Grid Computing Workshop, Denver, Colorado, USA (November 2001) 8. Abawajy, J.H.: Placement of File Replicas in Data Grid Environments. In: Bubak, M., van Albada, G.D., Sloot, P.M.A., Dongarra, J.J. (eds.) ICCS 2004. LNCS, vol. 3038, pp. 66– 73. Springer, Heidelberg (2004) 9. Rahman, R.M., Barker, K., Alhajj, R.: Replica Placement in Data Grid: A Multi-objective Approach. In: Zhuge, H., Fox, G.C. (eds.) GCC 2005. LNCS, vol. 3795, pp. 645–656. Springer, Heidelberg (2005) 10. Rahman, R.M., Barker, K., Alhajj, R.: Replica Placement Design with Static Optimality and Dynamic Maintainability. In: Proceedings of the Sixth IEEE Int’l Symposium on Cluster Computing and the Grid (CCGrid’06) (2006) 11. Raganathan, K., Lamnitchi, A., Foster, I.: Improving Data Availability through ModelDriven Replication for Large Peer-to-Peer Communities. In: proceedings of Global and Peer-to-Peer Computing on Large-Scale Distributed Systems Workship, Berlin, Germany, (May 2002) 12. Jin, H., Mao, F., Chen, H., Wu, S., Zou, D.: P2P Based Decentralized Dynamic Replica Placement Strategy in Data Grid Environment. Downloaded on October 2006 (2006) from http://grid.hust.edu.cn/fmao/papers/new/replica%20placement-0224IE%20fianl%20.pdf 13. Lamehamedi, H., Szymanski, B., Shentu, Z., Deelman, E.: Data Replication Strategies in Grid Environments. In: proceedings of 5th Int’l Conference on Algorithms and Architecture for Parallel Processing, pp. 378–383 (2002) 14. Wolski, R.: Forecasting Network Performance to Support Dynamic Scheduling Using the Network Weather Service. In: proc. 6th IEEE Symposium on High Performance Distributed Computing. Portland, Oregon (1997)

A Workload Balancing Based Approach to Discourage Free Riding in Peer-to-Peer Network* Yijiao Yu and Hai Jin Services Computing Technology and System Lab Cluster and Grid Computing Lab Huazhong University of Science and Technology, Wuhan, 430074, China [email protected], [email protected]

Abstract. We propose a workload balancing based approach to discourage free riding in Peer-to-Peer (P2P) network. The heart of our mechanism is to migrate some shared files from the overloaded peers to the neighbored free riders automatically and transparently, which enforces free riders to offer services when altruistic peers are heavily overloaded. The implementation techniques are discussed in detail. We design a simulation to verify this approach, and the results show that it can not only decrease the degree of free riding, but also improve the Quality of Service (QoS) and robustness of P2P network efficiently.

1 Introduction In the P2P philosophy, all peers in P2P network are volunteers, which should try to share files and offer services for other peers. In 2000, Adar et al. measured the Gnutella network, however, they found that only 10 percent peers provide almost 90 percent services in the P2P network [1]. Most of the peers share no more than 10 files or some unpopular files, which are not queried or downloaded by others at all. That is to say, most of the peers in the P2P network consume far more files than they offer. This selfish behavior of peers is called free riding and the selfish peers are regarded as free riders [1]. Papers [2][3] report the traffic measurement results of Gnutella in 2001 and 2005 respectively, which show that free riding has increased significantly since 2000. Moreover, free riding also prevails in Fast-Track, DirectConnect [4]. Free riding is a serious problem in P2P network for both ISPs and users. From an ISP’s point of view, free riding leads to the mismatch between the overlay network and the physical network, which makes the in-through traffic of physical network asymmetric to the out-through traffic of physical network [2]. To users, the response time of query increases significantly because a few altruistic peers have to handle too many requests. Especially, when a hot peer crashes due to heavy overload, the shared files of the whole P2P network will decrease sharply. Also, the topology of P2P *

This paper is supported by National Basic 973 Research Program of China under grant No.2003CB317003, and the Cultivation Fund of the Key Scientific and Technical Innovation Project, Ministry of Education of China under grant 705034.

K.C. Chang et al. (Eds.): APWeb/WAIM 2007 Ws, LNCS 4537, pp. 144–155, 2007. © Springer-Verlag Berlin Heidelberg 2007

A WBBA to Discourage Free Riding in P2P Network

145

network will change dramatically, and even the connected P2P network will be divided into multi-isolated networks [3][5]. In a word, the "tragedy of the commons" has now emerged, therefore the rational P2P application software developers should develop some anti-free riding measures to deal with it. Some mechanisms have been proposed to address free riding, such as incentive mechanisms, game theoretic approaches, economics or social network based methods. There is a general principle of the previous work, which is to decrease the QoS of free riders consuming. Unfortunately, most of users enjoy high QoS of download, but they do not contribute to the P2P network. ISPs are interested in the number of P2P users. If the number of users decreases sharply due to poor QoS, it is difficult to persuade ISPs to accept the anti-free riding measures. In addition, there are some guarded hosts behind of the Network Access Translation (NAT) or firewall [6], which cannot provide services to P2P network. A lot of peers cannot offer as much as they consume due to the asymmetric physical network connection, such as the ADSL. Therefore, the anti-free riding method should consider the real network environments and the benefits of both the ISPs and the users. The objective of this paper is to find an anti-free riding measure, which can not only guarantee the P2P networks run stably with high QoS but also attract a great many users. In addition, the novel approach should be easily implemented in practice. Therefore, a workload monitoring and balancing based approach is proposed and simulations are designed to verify it. On one hand, we expect all peers can enjoy high QoS of consuming, and the free riders are not punished too strictly. On the other hand, the P2P network should be robust and stable. The paper is organized as follows. Related work is discussed in section 2. In section 3, we present the overview of the workload balancing based approach, and analyze its specific features. The detailed implementation techniques of the mechanism are discussed in section 4. In section 5, we design a simulation to demonstrate the performance of our approach. Section 6 concludes this paper with a discussion of future work.

2 Related Work Ramaswamy et al. [5] made the first effort to avoid free riding in P2P network with the incentive mechanisms. Utility functions are proposed to evaluate the contribution of each peer, which are based on the number of shared files, the account of files size, and the contribution of upload. The QoS of each peer consuming is computed according to the utility functions. Some different utility functions and implementation protocols are defined in [7][8], which are more reasonable and complicated. For example, the contribution is not computed by the absolute size of upload, but the ratio between the traffic and the upload bandwidth of peer [7]. Since the utility functions and reputations are evaluated by the peer itself, the whitewashing behavior is employed by malicious peers. Ref. [9] addresses the dilemma about how to deal with whitewashing and gives some clues. Karakaya et al. [10] proposed a distributed and measurement-based framework against free riding in P2P networks. There is an important difference between [10] and [7][8] that every peer is measured and evaluated by neighbors. The requests from

146

Y. Yu and H. Jin

free rider will be discarded or with lower priorities, or even the link to free rider will be broken. The punishment is executed by neighbors, which can avoid the cheating behavior efficiently. Gupta et al. [11] used the game theory to analyze the action strategies of each peer. All peers on line are regarded as a set of players, and all players want to find the best action preferences for themselves. At the same time, the P2P network should get maximum benefits from the action strategies. A game theoretic approach to provide incentive and service differentiation in P2P Networks is presented in [12]. The analysis of the actions strategies are smart, however, they are difficult to be applied into engineering due to the high complexity and the requirement of the real time status information of the P2P network. Courcoubetis et al. [13] prove that more complex models are likely to lead to intractable game design problems and simple policies are asymptotically optimal. Krishnan et al. [14] analyze the impact of free riding to P2P network with the social network theory. They present that free riding is an instinctive behavior of human, and the P2P network designers should accept this behavior. In their opinion, a little free riding can improve the efficiency of public resources sharing. Economic models are employed in [15]-[17].

3 The Workload Balancing Based Approach In this section, we describe our mechanism, compare it to the previous mechanisms, and list its specific features. 3.1 Overview of the Novel Approach An important assumption in this paper is that the QoS of P2P network will decrease if the overloaded peers crash or exit the P2P network, because usually most of P2P connections and shared files are kept by overloaded peers. If they exit, a lot of peers will lose the connections to the P2P network, and even the whole P2P network becomes multi-isolated sub-networks. In our opinion, the protection of hub peers (peers which are extremely highly connected by others) is one of the most important concerns to sustain the P2P network and guarantee the QoS of all peers. To attract more users, free riding is allowed if there are no overloaded peers in the P2P network. If an overloaded peer appears, it decreases its workload as soon as possible to protect itself. The overloaded peers try to migrate some files to other neighbored free riders, and these free riders are enforced to offer services to the P2P network. Therefore, the motivation of anti-free riding is realized. It is notable that the negotiation of migration is executed through software automatically in the backend, and users are not aware of file migration. The report of status should be honestly, and the file receivers should provide services to other peers as soon as possible. The pseudo-code of workload balancing based approach (referred to as WBBA in the rest of this paper) to discourage free riding is illustrated as follows. It is clear that WBBA is a daemon thread, which starts as the peer joins in the P2P network, executes all the time, and ends until the peer exits. With the daemon thread, each peer can reduce overloaded workload to protect itself actively and automatically.

A WBBA to Discourage Free Riding in P2P Network

147

WBBA { Step1: Get the thresholds of overloaded status; Setp2: Monitor the peer’s status periodically; Step3: Judge whether the peer is overloaded. If (overloaded==true) goto step 4; else goto step9; Step4: Compute the set of destination peers to accept the migrated files Step5: Compute the set of migrated files; Step6: Assign migrated files to the destinations; Step7: Migrate shared files by FTP or HTTP; Step8: The overloaded peer updates the routing of the P2P network; Step9: Sleep a sample interval; goto setp2. } It should be noted that WBBA is executed in every peer and the overloaded thresholds are only about a peer. WBBA need not the global status information of the P2P networks. Connection control is a popular and efficient way to avoid free riding in P2P networks, e.g. [3][10], and it is also adopted in the Step 8 in WBBA. 3.2 The Specific Features of WBBA Compared WBBA to previous solutions, there are distinct differences. First, there are no utility functions and no limitations about the QoS a peer consumes. All peers are eager to get files and services from the P2P network, and all of them want to enjoy high QoS greedily. Only when the QoS meets most peers’ requirement, the number of peers will increase and the shared files will become abundant. The research of public files management has shown that a lot of users are of the selfish instinct. If users are required to know and select the policy of consuming and offering, it not only decreases the usability of software, but also leads some users to reject the P2P application. The benefits of ISPs will be damaged. In a word, with the novel mechanism, the P2P network tries its best efforts to provide services to all peers, both altruistic peers and selfish peers. Second, WBBA is a distributed algorithm and easily implemented in engineering. Each peer only focuses on its status of workload. Central monitor and global status information of P2P network are not required in WBBA. Moreover, each peer samples the status information at its sampling point, thus synchronization is not needed. The time complexity of monitoring is very low, which does not bring too much burden. Last but not least, the peers behind firewall and NAT can enjoy high QoS of download services. With the popular incentive mechanisms, these peers are regarded as free riders and offered poor download bandwidth. In fact, the special physical network environment forbids them to provide services to users outside. The P2P network tries its best to serve all peers. Therefore, the average QoS of all peers enjoy can be improved.

148

Y. Yu and H. Jin

4 The Implementation Techniques In this section, we will introduce the detail implementation techniques of WBBA. There are nine steps in WBBA. Some of them are easy to be realized, such as the Step7 and Step9. For simplicity, the implementation of Step7 and Step9 are not discussed in this paper. We categorize the other steps in WBBA into three issues, namely the status monitoring and judgment of overload, the selection of the migration destination peers, and selection and assignment of the migration files. 4.1 Status Monitoring and Judgment of Overload From Step1 to Step3, a peer monitors its real time workload status and judges whether it is overloaded and which status information should be monitored. From our perspective, three types of status information should be monitored and considered with high priorities, namely the P2P application resources type (P2PART), the hardware resources type (HRT), and the physical network resources type (PNRT). Table 1 lists the most necessary monitoring objects of the status. The left column lists the variable name, and the second column illustrates the type of the monitored information. The values of these variables are categorized into integer type and ratio type, which are shown in the right column. The variables listed in Table 1 are very important and necessary to WBBA, and P2P application designers can add other variables to satisfy with its special requirements. Table 1. The most necessary monitoring variables of the peer status Monitoring variables The number of files The access frequency The number of P2P connections The frequency of each file in a peer The number of forwarding P2P packages The CPU usage The memory usage The bandwidth usage The number of TCP active connections The number of TCP passive connections

Type P2PART P2PART P2PART P2PART P2PART HRT HRT PNRT PNRT PNRT

Value type Integer Integer Integer Integer Integer Ratio Ratio Ratio Integer Integer

The status of P2PART variables can be obtained with the P2P application directly, by adding some log operations, such as recording the access frequency of each file. The status of HRT variables can be obtained with Application Programming Interface (API) provided by the operating system. Due to the heterogeneity of APIs provided by different operating systems, the Simple Network Management Protocol (SNMP) can be utilized to sample both the HRT information and the PNRT information. Here, we do not discuss the detail implementation about the sampling. The definition of overloaded thresholds of monitoring variables is the premise of the judgment of overload. From Table 1, we can see that the threshold of ratio type variable is easier to be defined. For example, most of users do not expect the CPU

A WBBA to Discourage Free Riding in P2P Network

149

usage larger than 60%, otherwise the response of human-machine is very slow. Therefore, the default threshold of CPU usage is 60%. In this way, we can define all default thresholds of ratio type variables. However, due to the heterogeneity of software and hardware configuration of peers, the service abilities are distinct. The threshold of the integer type variables should be defined according to the configuration environments of each peer. Fortunately, either with the API or SNMP, we can get the hardware and software configuration information automatically. The daemon thread samples the status information periodically. There is no unique rule for all P2P applications because the focuses of different applications are different. For a P2P high performance computing system, CPU usage is the most important variable in the judgment rule. While in a file sharing P2P system, the bandwidth usage is the more important. A reasonable one is usually a combination of multiple variables listed in Table 1. In section 4.2 and 5.1, we illustrate two judgment rules. 4.2 The Selection of Migration Destination Peers When a peer detects itself being overloaded, it will migrate some shared files to the neighbored free riders immediately. The step4 selects the destination peers for migration. According to the measurement results in [1][2][4], there are hundreds or thousands of peers connecting with an altruistic peer. It is a combinatorial problem of selecting the appropriate destination peers from thousands of peers. There are a lot of strategies in selecting destination peers. For example, if the selection is based on fairness, we should select peers with the shortest queues. If the data access patterns are considered to improve the migration efficiency, we should copy the frequent accessed files in overloaded peer to free riders near the service requesters. To describe the main ideas of WBBA, we only illustrate a simple way, based on the number of connections of each peer, to select the destination peers. Most peers keep no more than 4 connections [2]. The number of connections indicates the number of services a peer provides and enjoys currently. If the number is too small, the peer is impossible to offer lots of services to other peers and the contribution to P2P network is also small. The number of connections is explicit and easily measured by each peer. Therefore, it is an appropriate measurement of the contribution of each peer and is used in the destination selection in this paper. Fig. 1 illustrates a topology of a local network in a large P2P system. Every peer is modeled as a node and the logic links between peers are modeled as undirected edges. There are 9 peers in Fig.1, marked as 1, 2, …, 9. The number of simultaneous connections of each peer is defined as the degree of the node in this paper. There are more than one link attached to peer 1, 2 and 3, and these three peers are easy to become the bottleneck of the P2P system. In this paper, altruistic peer means a special peer, which try to offer services for other peers in the P2P network. Overloaded peer is the peer whose workload is too heavy. Altruistic peer is easier to be overloaded, because it has too many workloads. However, not all altruistic peers are overloaded, if the altruistic peers are powerful enough.

150

Y. Yu and H. Jin Overloaded peer

1

Altruistic peer Free rider

3

2

4

5

6

7

8

9

Fig. 1. A topology of a P2P network

Suppose that peer 2 reports overload because too many peers connect to it in Step3. Then the Step4 is executed as follows by peer 2. First, peer 2 broadcasts the status query messages to all neighbored peers, namely peer 1, 3, 4, 5, 6, and 7. Second, all neighbors reply the query message and report their latest status information. Third, peer 2 receives these feedback messages and judges who will be the destination peer. Given that the migration of shared files also costs a lot of computer resources and network bandwidth, we do not encourage the frequent migration of shared files between peers. Especially, the migration between busy peers and overloaded peers cannot reduce the service overload, and even makes the network become unstable. We define a simple rule for this case, the degree of destination node is no more than half of the degree of the overloaded node. In Fig.1, the degree of peer 2 is 6, and the degree of peer 3 is 4. Therefore, peer 3 is regarded as busy and will not be selected as the destination peer. We define the maximum set of destination peers (MSDP), in which all elements comply with the above rule, e.g. MSDP of peer 2 is {1, 4, 5, 6, 7}. The number of final destination peers, marked as m (1 ≤ m ≤ |MSDP|), is generated randomly. Finally, select m different peers from MSDP randomly. For instance, m is 3 and {4, 5, 6} is selected as the final destination peers in Fig.1. We should explain why we do not select the neighbors with the least connections as the destination peer. In a large P2P network, the degree of hub nodes can reach at more than 20,000 or 30,000. If the number of elements in MSDP is too large, sorting will cost a lot of time, which will delay the procedure of decreasing workload. Generating m different random numbers are very quick and does not cost a lot of memory resources. Using MSDP as the final set of destination peers is unacceptable in engineering, because physical network will be congested if sending too many files to many neighbored peers in a short time. We suggest that m should be no more than 50 in real environments. It should be noted that the overloaded peer should stop some most frequently accessed files services immediately when either all neighbored peers are busy, or it cannot find an appropriate destination peer to migrate files. 4.3 The Selection and Assignment of Migration Files After the set of destination peers is chosen, Step5 will be executed. According to the measurement results in [1][3], there are thousands of files shared by altruistic peers and the size of them is also very large. It is unnecessary and impossible to copy all

A WBBA to Discourage Free Riding in P2P Network

151

shared files to other peers. The basic principle of the migration is to decrease the workload sharply by transferring files as few as possible. There is an efficient and feasible way applied in our simulation, which only transfers the most frequently accessed shared files to free riders. To realize it, the P2P application system should record the access frequency of each file. Fortunately, this requirement is not difficult to be realized in engineering, and some P2P applications have provided this function. We only need to sort the most frequent files from large to small, and get the top n files to be migrated (suppose n (m ≤ n) files will be migrated). There are mn kinds of different assignments solutions of migration files. For instance, if peer 2 wants to migrate the top 4 files to the three destination peers in Fig. 1, there are 34 (81) kinds of assignment solutions. The number of assignment solutions increases exponentially with n. Researchers are interested in the optimal solution among the mn kinds of solutions. However, we do not encourage spending too much time on looking for the optimal solution. Assigning the most frequently accessed files to the peers with least connections in the set of destination peers is an approximate optimal assignment in the simulation. After the assignment of migration, the overloaded peer migrates shared files to free riders with FTP or HTTP protocols. There may be problems in the middle of the file migration process, since the network may be unreliable or the file may be too large. In this case, both the migration and the services of the unsuccessfully migrated files in the source peer should be stopped. When the migration is finished, the overloaded peer updates Distributed Hash Table (DHT) or routing table immediately. When the new requirements to the migration files come, it only forwards the requirement to the destination peer and stops download service.

5 Experiments In this section, we design a simulation to illustrate the performance of WBBA. 5.1 Simulation Setup All simulation programs are designed with Java and executed in a personal computer, with 1.7GHz Intel processor and 256MB memory. In the simulation, the first task is to make a virtual P2P network environment similar to the real P2P network environment. Especially, the network topology, the distribution of the number of files shared by each peer, and the distribution of access frequency of every file in each peer must comply with that of the real P2P network. The Pajek software [18] is employed in our simulation to obtain network topology with the scale free feature, which can generate different kinds of network topology, such as random network, scale free and Erdos-Renyi. Some peers generated by Pajek have no connections with other peers. From our perspective, all peers should connect to the P2P network. Otherwise, they cannot get services from the P2P network. Therefore, the original topology generated by Pajek is preprocessed to make it to be a connected graph. In real P2P environments, some peers join and depart frequently. However, it is not easy to describe the probability of peer’s dynamic behaviors, and in many simulations the

152

Y. Yu and H. Jin

dynamic behaviors are not discussed [5]. Thus, join and departure of peers are not considered in this paper. The number of files shared by peer i is generated with Equation (1), where di is the degree (the number of simultaneous connections) of peer i. The function random(n) generates a pseudorandom, uniformly distributed integer value between 0 and n-1. In Equation (1), a is a constant, which represents the degree of cooperation of peers. In our simulation, a is set as 3. Since the network topology is generated with the scale free network model, the distribution of degrees is similar to the real network. The previous measurement results show that peers with large connections usually share more files. In general, Equation (1) can generate large number of shared files for these peers with large degrees. However, if the degree of peer i is large and the number of files shared is small, peer i will be regarded as routing peers in P2P network. These routing peers devote to forwarding P2P packages as the router in Internet. Equation (1) is only executed once for a peer while initializing the network environment.

sharedFile si = 1 + random (a × d i )

(1)

The accessed frequency in the tth iteration of the jth file in peer i is generated with Equation (2), where b is a nonnegative constant representing the popularity of files in P2P system. di,t represents the degree i in the tth iteration. It is clear that the access frequency in the tth iteration of the jth file in peer i is related to that of the t-1th iteration, and c is a constant between 0 and 1. This assumption is reasonable in real P2P network. In our simulation, c is set as 0.6 and b is set as 20. In every iteration period, the access frequency of every files will be computed with Equation (2). f i , j ,t = c × f i , j ,t −1 + (1 − c) × random(b × d i ,t )

(2)

In the simulation, we compute the workload of each peer only with some P2PART variables. Equation (3) is defined to compute the workload of peer i in the tth iteration, which is based on the access frequencies, file size and the degree of peer. Due to the overloaded peer updates routing table, the degree of peer i is changed with t. sizei,j is the file size of the jth file in peer i. fi,j,t is the access frequency of the jth file in peer i in the tth interval. workloadi,t denotes the average traffic of each link in the tth iteration. sharedFilesi is also changed in different iteration periods, because the shared files are migrated from the overloaded peer to free riders. After the migration of shared files, sharedFiles of the overloaded peer will be decreased by 1 and that of the destination peer will be increased by 1. sharedFilesi , t

workload i ,t =

∑f i =1

i , j ,t

× sizei , j

(3)

d i ,t

Moreover, considering the heterogeneity of computers, two overloaded thresholds of workloads are used in simulation to make the judgment of overload fair and similar to the real environment to represent computers of different processing capabilities, which are 20000 and 4000. If workloadi,t is larger than the threshold of each peer, peer i will be regarded as overloaded.

A WBBA to Discourage Free Riding in P2P Network

153

5.2 Simulation Results We evaluate the performance of the novel anti-free riding measure with three aspects, including the distribution of the degree of each peer, the distribution of the number of files shared by each peer, and the distribution of the access frequency of each peer. In addition, we are interested in how many peers have to do the migration operation, and how long the P2P network will be stable and robust. There are 2000 peers and 7827 logic links in the network topology graph. The detailed results are illustrated as follows. It is notable that lg(x) in all figures in this paper denotes the base 10 logarithm. Fig. 2 illustrates the distribution of the degree of each peer. In Fig. 2, the largest degree of peers is no more than 105, and the smallest one is 1. Due to the heterogeneity of degrees, the numOfPeers is accounted with the step of 5. For instance, all peers with the degree between 1 and 5 are accounted as numOfPeers at degree = 5, and numOfPeers at degree = 10 is the number of peers whose degree is between 6 and 10. It is clear that the number of peers, whose degree is no more than 10, decreases after the anti-free riding control, and some peers have more neighbors. Moreover, the number of hub peers, with very high degrees, declines. 3.5

3 After control Before control

After control 3

Before control

2.5 Free riders

lg(numOfPeers)

2 Altruistic peers

1.5

Hub peers

1

Altruistic peers

2 1.5 1

Hub peers 0.5

0.5 0 0

lg(numOfPeers)

Free riders 2.5

10 20

40 60 80 Degree of peers

100

Fig. 2. Distribution of the degrees

120

0 0.5

1

1.5 lg(sharedFiles)

2

2.5

Fig. 3. Distribution of the number of shared files

The distribution of the number of files shared by each peer is shown in Fig.3. The dot line illustrates the distribution status before anti-free riding control, and the solid line describes the distribution after control. It is clear that the number of free riders, sharing no more than 10 files, decreases, and more peers share more than 10 files. However, the shared files are no longer in several hub peers, because these files in overloaded peers are migrated to neighbored free riders. From Fig.3, we can see that the free riding phenomenon is discouraged, and the overloaded peers or hub peers are removed successfully. Fig. 4 illustrates the distribution of the access frequency of each peer during an iteration period. We use 100 as a basic step to compute numOfPeers. For example, all peers, the access frequency between 101 and 200 in an iteration period, are accounted as the numOfPeers of frequency=200. The dot line is the distribution of the frequencies of peers before control, and the solid line is the distribution of frequencies after control. The number of peers, offering service no more than 300 times in an

154

Y. Yu and H. Jin

iteration period, decreases, and the number of altruistic peers with frequency between 301 and 7,000 increases. Moreover, the largest access frequency decreases sharply after the control, which shows the altruistic peers provide more services. 3

numOfOverloadedPeers

2.5 lg(numOfPeers)

20

After control Before control Altruistic peers

2 Free riders 1.5 1

Hub peers 0.5 0 2

frequency=300

2.5

3

3.5 4 lg(frequency)

4.5

5

Fig. 4. Distribution of the accessed frequency of peers

15

10

5

0 0

5

10 Iterations

15

20

Fig. 5. The number of overloaded peers

We are interested in the time to get rid of all overloaded peers in a P2P network. Fig. 5 shows the changes in the number of overloaded peers over the iteration from 0th to 19th in the simulation. There is no overloaded peer after 19 iterations and the P2P network becomes robust. The maximum number of overloaded peers is 20, which is no more than 1% of the number of all peers in the P2P network. It illustrates that only few peers in P2P network execute the migration operation, which is feasible to practical applications. The phenomenon is also similar to the real P2P network environment that about 1% peers are very busy to provide 90% services. The number of overloaded peers does not always decrease with the increase of iterations, because new requests come to each peer in every iteration period.

6 Conclusion In this paper, we present a workload balancing based anti-free riding approach, which is easily implemented in engineering. The simulation results show that the simple way suffices to control of the overall system to a nearly optimal objective, in which the P2P network is sustained as a connected graph and all peers can consume QoS as high as possible. The most specific feature of this way is that it not only discourages free riding behaviors of peers strictly, but also guarantees the QoS of all peers, including the free riders and guarded peers behind firewall and NAT. This feature attracts a great many peer users and is helpful to maintain the business model. There are many ways to extend the approach in the future. For instance, we might discuss the selection strategies of the destination peers to migrate files, especially when there are too many free riders connecting to the overloaded peers. Different strategies should be considered, such as the fairness, and the data access pattern. Moreover, it is desired to discuss the impacts of join and departure behaviors of peers.

A WBBA to Discourage Free Riding in P2P Network

155

References 1. Adar, E., Huberman, B.: Free Riding on Gnutella. First Monday, 10 (2000) 2. Ripeanu, M., Foster, I., Iamnitchi, A.: Mapping the Gnutella Network. IEEE Internet Computing 1, 50–57 (2002) 3. Hughes, D., Coulson, G., Walkerdine, J.: Free Riding on Gnutella Revisited: The Bell Tolls? IEEE Distributed Systems Online, 6 (2005) 4. Sen, S., Wang, J.: Analyzing Peer-to-Peer Traffic Across Large Networks. IEEE/ACM Transactions on Networking 2, 219–232 (2004) 5. Ramaswamy, L., Liu, L.: Free Riding: A New Challenge to Peer-to-Peer File Sharing Systems. In: Proceeding of the 36th Hawaii International Conference on System Sciences, 220-229 (2003) 6. Wang, W.J., Chang, H., Zeitoun, A., Jamin, S.: Characterizing Guarded Hosts in Peer-toPeer File Sharing Systems. In: Proceedings of Globecom 1539–1543 (2004) 7. Ma, R.T.B., Lee, S.C.M., Lui, J.C.S., Yau, D.K.Y.: An Incentive Mechanism for P2P Networks. In: Proceedings of the 24th International Conference on Distributed Computing Systems, 516–523 (2004) 8. Anceaume, E., Gradinariu, M., Ravoajia, A.: Incentive for P2P Fair Resource Sharing. In: Proceedings of the Fifth IEEE International Conference on Peer-to-Peer Computing, 253–260 (2005) 9. Feldman, M., Papadimitriou, C., Chuang, J., Stoica, I.: Free-Riding and Whitewashing in Peer-to-Peer Systems. IEEE Journal of Selected Areas in Communications 5, 1010–1019 (2006) 10. Karakaya, M., Korpeouglu, I., Ulusoy, O.: A Distributed and Measurement-based Framework Against Free Riding in Peer-to-Peer Networks. In: Proceedings of the Fourth International Conference on Peer-to-Peer Computing, 276–277 (2004) 11. Gupta, R., Somani, A.K.: Game Theory as a Tool to Strategize As Well As Predict Nodes Behavior in Peer-to-Peer Networks. In: Proceedings of the 11th International Conference on Parallel and Distributed Systems, 244–249 (2005) 12. Ma, R.T.B., Lee, S.C.M., Lui, J.C.S., Yau, D.K.Y.: A Game Theoretic Approach to Provide Incentive and Service Differentiation in P2P Networks. In: Proceedings of SIGMETRICS/ Performance’04, 189–198 (2004) 13. Courcoubetis, C., Weber, R.: Incentive for Large Peer-to-Peer Systems. IEEE Journal of Selected Areas in Communications 5, 1034–1050 (2006) 14. Krishnan, R., Smith, M.D., Tang, Z.L., Telang, R.: The Impact of Free-Riding on Peer-toPeer Networks. In: Proceeding of the 37th Hawaii International Conference on System Sciences, 199–208 (2004) 15. Strulo, B., Smith, A., Farr, J.: An Architecture for Peer-to-Peer Economies. In: Proceedings of the Third International Conference on Peer-to-Peer Computing, 208–209 (2003) 16. Gupta, R., Somani, A.K.: Pricing Strategy for Incentive Selfish Nodes to Share Resources in Peer-to-Peer (P2P) Networks. In: Proceedings of the 12th IEEE International Conference on Networks, 624–629 (2004) 17. Ham, M.J., Agha, G.: ARA: A Robust Audit to Prevent Free-Riding in P2P Networks. In: Proceedings of the Fifth IEEE International Conference on Peer-to-Peer Computing, 125–132 (2005) 18. Pajek, Available at http://vlado.fmf.uni-lj.si/pub/networks/pajek/

QoS-Based Services Selecting and Optimizing Algorithms on Grid Qing Zhu, Shan Wang, Guorong Li, Guangqiang Liu, and Xiaoyong Du School of Information, Renmin University of China, Beijing 100872, P.R. China Key Laboratory of Data Engineering and Knowledge Engineering (Renmin University of China), Beijing 100872, P.R. China [email protected]

Abstract. QoS-Based Services Selecting and Optimizing Composition between the peers play an increasingly important role to ensure interoperability on Grid environment. However, the prohibitive cost of selecting, matching, mapping and composing algorithm has now become a key bottleneck hindering the deployment of a wide variety of Grid services. In this paper, we present QoS-Based Services Selecting and Optimizing Composition on Grid. First, it checks requesters’ semantic in order to form candidate service graph. Second, it designs service selecting and mapping algorithms for optimizing the model. Third, it creates an executed plan of optimum composition on Grid. We conducted experiments to simulate and evaluate our approach. Keywords: Information