Integrated Management of Systems, Services, Processes and People in IT: 20th IFIP IEEE International Workshop on Distributed Systems: Operations and ... Networks and Telecommunications)

Lecture Notes in Computer Science Commenced Publication in 1973 Founding and Former Series Editors: Gerhard Goos, Juris ...

Author: Claudio Bartolini | Luciano Paschoal Gaspary

17 downloads 758 Views 4MB Size Report

This content was uploaded by our users and we assume good faith they have the permission to share this book. If you own the copyright to this book and it is wrongfully on our website, we offer a simple DMCA procedure to remove your content from our site. Start by pressing the button below!

Report copyright / DMCA form

DOWNLOAD PDF

Lecture Notes in Computer Science Commenced Publication in 1973 Founding and Former Series Editors: Gerhard Goos, Juris Hartmanis, and Jan van Leeuwen

Editorial Board David Hutchison Lancaster University, UK Takeo Kanade Carnegie Mellon University, Pittsburgh, PA, USA Josef Kittler University of Surrey, Guildford, UK Jon M. Kleinberg Cornell University, Ithaca, NY, USA Alfred Kobsa University of California, Irvine, CA, USA Friedemann Mattern ETH Zurich, Switzerland John C. Mitchell Stanford University, CA, USA Moni Naor Weizmann Institute of Science, Rehovot, Israel Oscar Nierstrasz University of Bern, Switzerland C. Pandu Rangan Indian Institute of Technology, Madras, India Bernhard Steffen University of Dortmund, Germany Madhu Sudan Microsoft Research, Cambridge, MA, USA Demetri Terzopoulos University of California, Los Angeles, CA, USA Doug Tygar University of California, Berkeley, CA, USA Gerhard Weikum Max-Planck Institute of Computer Science, Saarbruecken, Germany

5841

Claudio Bartolini Luciano Paschoal Gaspary (Eds.)

Integrated Management of Systems, Services, Processes and People in IT 20th IFIP/IEEE International Workshop on Distributed Systems: Operations and Management, DSOM 2009 Venice, Italy, October 27-28, 2009 Proceedings

13

Volume Editors Claudio Bartolini Hewlett-Packard Laboratories 1501 Page Mill Road Palo Alto, CA, 94304, USA E-mail: [email protected] Luciano Paschoal Gaspary Federal University of Rio Grande do Sul (UFRGS) Institute of Informatics Caixa Postal 15064 91501-970 Porto Alegre, Brazil E-mail: [email protected]

Library of Congress Control Number: 2009936144 CR Subject Classification (1998): C.2, C.2.5, C.2.6, D.4.6, K.6.5, K.4.2, B.8 LNCS Sublibrary: SL 5 – Computer Communication Networks and Telecommunications ISSN ISBN-10 ISBN-13

0302-9743 3-642-04988-5 Springer Berlin Heidelberg New York 978-3-642-04988-0 Springer Berlin Heidelberg New York

This work is subject to copyright. All rights are reserved, whether the whole or part of the material is concerned, specifically the rights of translation, reprinting, re-use of illustrations, recitation, broadcasting, reproduction on microfilms or in any other way, and storage in data banks. Duplication of this publication or parts thereof is permitted only under the provisions of the German Copyright Law of September 9, 1965, in its current version, and permission for use must always be obtained from Springer. Violations are liable to prosecution under the German Copyright Law. springer.com © IFIP International Federation for Information Processing 2009 Printed in Germany Typesetting: Camera-ready by author, data conversion by Scientific Publishing Services, Chennai, India Printed on acid-free paper SPIN: 12775781 06/3180 543210

Preface

This volume of the Lecture Notes in Computer Science series contains all papers accepted for presentation at the 20th IFIP/IEEE International Workshop on Distributed Systems: Operations and Management (DSOM 2009), which was held in Venice, Italy, during October 27-28, 2009. DSOM 2009 was the 20th event in a series of annual workshops. It followed in the footsteps of previous successful meetings, the most recent of which were held on Samos, Greece (DSOM 2008), San Jos´e, California, USA (DSOM 2007), Dublin, Ireland (DSOM 2006), Barcelona, Spain (DSOM 2005), and Davis, California, USA (DSOM 2004). The goal of the DSOM workshops is to bring together researchers from industry and academia working in the areas of networks, systems, and service management, to discuss recent advances and foster future growth. In contrast to the larger management conferences, such as IM (International Symposium on Integrated Network Management) and NOMS (Network Operations and Management Symposium), DSOM workshops have a single-track program in order to stimulate more intense interaction among participants. The theme of DSOM 2009 was “Integrated Management of Systems, Services, Processes and People in IT,” keeping its focus on the traditional network and system management disciplines, as well as extending it onto the emerging research ﬁeld of managing all other “soft” aspects of IT, involving people and processes. IT management has evolved signiﬁcantly over the past few years as IT-based solutions have become increasingly critical to the functioning of organizations. From device, network, and systems management, solutions have evolved to include service management and IT governance, as witnessed by the recent popularity of ITIL and COBIT frameworks. As a result of this extended focus, DSOM 2009 included sessions on monitoring and analysis tools, service management, autonomic and self-management, people in IT and decision support for IT management, as well as on security and virtualization. Like the previous four DSOM workshops, DSOM 2009 was co-located with several related events as part of the 5th International Week on Management of Networks and Services (Manweek 2009). The other events were the 12th IFIP/IEEE International Conference on Management of Multimedia and Mobile Networks and Services (MMNS 2009), the 9th IEEE International Workshop on IP Operations and Management (IPOM 2009), the 4th IEEE International Workshop on Modeling Autonomic Communications Environments (MACE 2009), and the 6th International Workshop on Next Generation Networking Middleware (NGMN 2009). Co-locating these events provided the opportunity for an exchange of ideas between research communities that work on related topics, allowing participants to forge links and exploit synergies. DSOM 2009 attracted a total of 50 paper submissions by authors from 20 diﬀerent countries. Each paper received at least three, and in most cases four,

VI

Preface

reviews by experts in the ﬁeld. A total of 14 submissions were accepted into the program as full papers (acceptance rate of 28%). DSOM 2009 owes its success in large part to a dedicated community of researchers from academia and industry that has formed over many years. First and foremost, we want to thank the authors of the submitted papers – without them, there would be no program. We also want to thank the members of the Technical Program Committee and the additional reviewers for their constructive and detailed reviews. A big “thank you” goes to Tom Pfeifer and Alberto Gonzalez Prieto, our Publication Advisor and Chair, and the Springer LNCS team, who played a big part in creating these proceedings. Finally, we want to thank our patron, Telecom Italia, whose ﬁnancial support was essential to making DSOM 2009 a great event. October 2009

Claudio Bartolini Luciano Paschoal Gaspary

DSOM 2009 Organization

Workshop and Program Co-chairs Claudio Bartolini Luciano Paschoal Gaspary

Hewlett Packard Labs, USA Federal University of Rio Grande do Sul (UFRGS), Brazil

Publication Advisory Chair Tom Pfeifer

Waterford Institute of Technology, Ireland

Publication Chair Alberto Gonzalez Prieto

Royal Institute of Technology (KTH), Sweden

Finance Chair Raouf Boutaba

University of Waterloo, Canada

Infrastructure Chair Sven van der Meer

Waterford Institute of Technology, Ireland

Local Arrangements Chair Massimo Foscato

Telecom Italia Labs, Italy

Registration Chair Idilio Drago

University of Twente, The Netherlands

Publicity Co-chair Carlos Becker Westphall

Federal University of Santa Catarina (UFSC), Brazil

VIII

Organization

Manweek 2009 General Co-chairs Aiko Pras Roberto Saracco

University of Twente, The Netherlands Telecom Italia Labs, Italy

Manweek 2009 Advisors Raouf Boutaba James Hong Aiko Pras

University of Waterloo, Canada POSTECH, Korea University of Twente, The Netherlands

DSOM 2009 Technical Program Committee Issam Aib Ehab Al-Shaer Javier Baliosian Claudio Bartolini Nevil Brownlee Marcus Brunner Omar Cherkaoui Alexander Clemm Yixin Diao Metin Feridun Olivier Festor Luciano Paschoal Gaspary Kurt Geihs Yacine Ghamri-Doudane Lisandro Zambenedetti Granville Masum Hasan Heinz-Gerd Hegering Joseph Hellerstein James Hong Cynthia Hood Brendan Jennings Alexander Keller Wolfgang Kellerer Yoshiaki Kiriha George Kormentzas David Lewis Hong Li Antonio Liotta Emil Lupu Hanan Lutﬁyya

University of Waterloo, Canada DePaul University, USA University of the Republic, Uruguay Hewlett Packard Labs, USA The University of Auckland, New Zeland NEC Europe Ltd., Germany University of Quebec in Montreal, Canada Cisco Systems, USA IBM Research, USA IBM Research, Switzerland INRIA Nancy - Grand Est, France UFRGS, Brazil Universit¨ at Kassel, Germany LRSM - ENSIIE, France UFRGS, Brazil Cisco Systems, USA Leibniz Supercomputing Center, Germany Google Inc., USA POSTECH, Korea Illinois Institute of Technology, USA Waterford Institute of Technology, Ireland IBM Global Technology Services, USA DOCOMO Euro-Labs, Germany NICT, Japan University of the Aegean, Greece Trinity College Dublin, Ireland Intel Corporation, USA Eindhoven University of Technology, The Netherlands Imperial College London, UK University of Western Ontario, Canada

Organization

Jean-Philippe Martin-Flatin Sven van der Meer Maitreya Natu Saverio Niccolini Jos´e Marcos Nogueira Declan O’Sullivan Aiko Pras Alberto Gonzalez Prieto Juergen Quittek Danny Raz Gabi Dreo Rodosek J¨ urgen Sch¨ onw¨alder Joan Serrat Adarsh Sethi Morris Sloman Radu State Burkhard Stiller John Strassner Mauro Tortonesi David Trastour Filip De Turck Jorge L´opez de Vergara Vincent Wade Carlos Becker Westphall Geoﬀrey Xie Makoto Yoshida Xiaoyun Zhu

NetExpert, Switzerland Waterford Institute of Technology, Ireland Tata Research Development and Design Centre, USA NEC Europe Ltd., Germany UFMG, Brazil Trinity College Dublin, Ireland University of Twente, The Netherlands KTH, Sweden NEC Europe Ltd., Germany Technion, Israel University of Federal Armed Forces, Germany Jacobs University Bremen, Germany Universitat Polit`ecnica de Catalunya, Spain University of Delaware, USA Imperial College London, UK LORIA - INRIA Lorraine, France University of Zurich and ETH Zurich, Switzerland MDAPCE.com, USA University of Ferrara, Italy Hewlett Packard Labs, UK Ghent University, Belgium Universidad Aut´ onoma de Madrid, Spain Trinity College Dublin, Ireland UFSC, Brazil Naval Postgraduate School, USA The University of Tokyo, Japan VMware Inc., USA

Additional Reviewers Majed Alhaisoni Pieter-Tjerk de Boer Elisa Boschi Lei Chai Laurent Ciarletta Diana Comes Idilio Drago Jo˜ ao Gir˜ ao Felipe Huici Georgios Karagiannis Max Machado Felipe Mata

IX

University of Essex, UK University of Twente, The Netherlands Hitachi Europe Ltd., Switzerland VMware Inc., USA INRIA-LORIA Lorraine, France Universit¨at Kassel, Germany University of Twente, The Netherlands NEC Europe Ltd., Germany NEC Europe Ltd., Germany University of Twente, The Netherlands UFMG, Brazil Universidad Aut´ onoma de Madrid, Spain

X

Organization

Vlado Menkovski Nadia Nawaz Qadri Pedro Santiago del Rio Markus Schmid Hendrik Skubch Christoph Sorge Tim Stevens Yongning Tang Ha Manh Tran Iyad Tumar Enrique V´ azquez Bart De Vleeschauwer

Eindhoven University of Technology, The Netherlands University of Essex, UK Universidad Aut´ onoma de Madrid, Spain Fachhochschule Wiesbaden - University of Applied Sciences, Germany Universit¨ at Kassel, Germany NEC Europe Ltd., Germany Ghent University, Belgium Illinois State University, USA Jacobs University Bremen, Germany Jacobs University Bremen, Germany Universidad Polit´ecnica de Madrid, Spain Ghent University, Belgium

Table of Contents

Monitoring and Analysis Tools Design and Implementation of a Distributed Platform for Sharing IP Flow Records . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Cristian Morariu, Peter Racz, and Burkhard Stiller Design of a Stream-Based IP Flow Record Query Language . . . . . . . . . . . Vladislav Marinov and J¨ urgen Sch¨ onw¨ alder Monitoring Multiple Concurrent Service Level Parameters with Multidimensional Trees . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Andreas Kiefer, Elias P. Duarte Jr., and Cristina D. Murta

1 15

29

Service Management A Performance-Based Methodology to Improve Grid Exploitation . . . . . . A. Clematis, A. Corana, D. D’Agostino, A. Galizia, and A. Quarati

43

One Is Not Enough: A Hybrid Approach for IT Change Planning . . . . . . Sebastian Hagen, Nigel Edwards, Lawrence Wilcock, Johannes Kirschnick, and Jerry Rolia

56

Improving IT Change Management Processes with Automated Risk Assessment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Juliano Araujo Wickboldt, Lu´ıs Armando Bianchin, Roben Castagna Lunardi, Fabr´ıcio Girardi Andreis, Weverton Luis da Costa Cordeiro, Cristiano Bonato Both, Lisandro Zambenedetti Granville, Luciano Paschoal Gaspary, David Trastour, and Claudio Bartolini

71

Autonomic and Self-management A2A: An Architecture for Autonomic Management Coordination . . . . . . . Alexander V. Konstantinou and Yechiam Yemini Consistency of States of Management Data in P2P-Based Autonomic Network Management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . J´eferson Campos Nobre and Lisandro Zambenedetti Granville Self-optimizing Hybrid Routing in Publish/Subscribe Systems . . . . . . . . . Arnd Schr¨ oter, Daniel Graﬀ, Gero M¨ uhl, Jan Richling, and Helge Parzyjegla

85

99 111

XII

Table of Contents

People in IT and Decision Support for IT Management Workload Management in Dynamic IT Service Delivery Organizations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Yixin Diao and Aliza Heching ChangeAdvisor: A Solution to Support Alignment of IT Change Design with Business Objectives/Constraints . . . . . . . . . . . . . . . . . . . . . . . . Roben Castagna Lunardi, Weverton Luis da Costa Cordeiro, Fabr´ıcio Girardi Andreis, Juliano Araujo Wickboldt, Cristiano Bonato Both, Luciano Paschoal Gaspary, Lisandro Zambenedetti Granville, David Trastour, and Claudio Bartolini Towards Robust IT Service Portfolio Management . . . . . . . . . . . . . . . . . . . David Trastour and Athena Christodoulou

123

138

152

Security and Virtualization Hidden Markov Model Modeling of SSH Brute-Force Attacks . . . . . . . . . . Anna Sperotto, Ramin Sadre, Pieter-Tjerk de Boer, and Aiko Pras

164

Network Virtualization in Future Home Environments . . . . . . . . . . . . . . . . Andreas Berl, Roman Weidlich, Michael Schrank, Helmut Hlavacs, and Hermann de Meer

177

Author Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

191

Design and Implementation of a Distributed Platform for Sharing IP Flow Records Cristian Morariu, Peter Racz, and Burkhard Stiller Department of Informatics IFI, University of Zurich, Switzerland

Abstract. Experiments using real traffic traces are of key importance in many network management research fields, such as traffic characterization, intrusion detection, and accounting. Access to such traces is often restricted due to privacy issues; research institutions typically have to sign non-disclosure agreements before accessing such traces from a network operator. Having such restrictions, researchers rarely have more than one source of traffic traces on which to run and validate their experiments. Therefore, this paper develops a Distributed Platform for Sharing IP Flows (DipSIF) based on NetFlow records between multiple institutions. It is assumed that NetFlow traces collected by each participant are archived on separate storage hosts within their premises and then made available to others using a server that acts as a gateway to the storage. Due to privacy reasons the platform presented here uses a prefix-preserving, cryptography-based, and consistent anonymization algorithm in order to comply to different regulations determining the exchange of traffic traces data.

1

Introduction and Problem Statement

Internet Protocol (IP) traffic traces are widely used by researchers for different purposes. On one hand, traces are used to observe traffic characteristics to improve traffic analysis tools. On the other hand, they are used to evaluate new intrusion or traffic anomaly detection algorithms based on real traffic traces. One important drawback when designing new algorithms for the analysis of IP traffic is the lack of access to many real traffic traces based on which the new algorithms can be evaluated. Often traffic traces are used, which are available locally (such as those captured within a lab network or within a subnetwork of a university), but there is rarely access to traces collected by other parties. Problems that researchers face when accessing remote traffic traces are of both technical and legal nature. Technical problems relate mainly to the lack of common ways to access different repositories as well as the lack of a way to pre-process traffic traces before being downloaded, which often leads to downloading huge traces while only a small part of those traces is used for analysis. The legal problems that make access to flow traces difficult mainly relate to privacy and data protection laws. Although the authors of this paper agree that legal problems are very important to be solved in order to successfully deploy a NetFlow traces sharing platform, this paper only investigates technical problems that make sharing of NetFlow traces difficult. C. Bartolini and L.P. Gaspary (Eds.): DSOM 2009, LNCS 5841, pp. 1–14, 2009. © IFIP International Federation for Information Processing 2009

2

C. Morariu, P. Racz, and B. Stiller

This paper presents the Distributed Platform for Sharing IP Flows (DipSIF) that was developed in one of the activities of the European project EMANICS [7]. DipSIF can be used to share NetFlow records collected at multiple sites in order to make them available to researchers at different sites. In order to achieve this sharing, the platform offers well-defined interfaces allowing researchers to access in a similar way NetFlow records stored locally as well as remotely from any host connected to the Internet. In order to fulfil privacy protection and to comply to regulations for traffic trace exchange, the platform uses a prefix-preserving, cryptography-based, and consistent anonymization algorithm. The platform is fully specified, it has been implemented as a prototype as described below, and tested. After discussing related work in Sect. 2, the design of the IP flow records sharing platform is presented in Sect. 3. While Sect. 4 outlines implementation details, Sect. 5 provides a functional evaluation of the system. Finally, Sect. 6 summarizes the paper and discusses possible future work.

2

Related Work

The two most important aspects that determine the design of DipSIF are (a) the traffic trace collection and storing as well as (b) the anonymization for sharing. 2.1

Flow Record Collection and Storage

As defined in [5] a flow record contains information about a specific flow that was observed at an Observation Point. A flow record contains measured properties of the flow (e.g., the total number of packets and bytes transferred in the flow) and usually characteristic properties of the flow (e.g., source and destination IP addresses). Typically an IP flow record is identified by at least the following unique keys: source IP address, destination IP address, source port, destination port, and protocol type. Different solutions, tools, and protocols exist to collect and store IP flow records. etFlow [4] — embedded in most Cisco routers — includes a NetFlow exporter component, which summarizes the traffic traversing routers in NetFlow records and exports those to remote collectors using the NetFlow protocol. Today different versions of the NetFlow protocol exist. Version 1 — the export format released initially — is rarely used today. Version 5 enhanced the initial specification and it is the most commonly used version today. Version 7 added support for data export from aggregation caches. The latest version is Version 9 and it specifies a flexible and extensible format, which provides the versatility needed for supporting new fields and record types. This format accommodates support of new technologies and introduces a template-based flow record transfer. Templates provide an efficient record transfer and allow record format extensions without concurrent changes to the basic record format. The IP Information Export (IPFIX) protocol [5] is a flow record transfer protocol standardized by the Internet Engineering Task Force (IETF). The IPFIX protocol is based on the NetFlow v9 protocol.

Design and Implementation of Distributed Platform for Sharing IP Flow Records

3

Cflowd [3] is a flow analysis tool that can be used to analyze Cisco NetFlow records. The package allows data collection as well as analysis in support of trends estimation, traffic tracking, accounting, and billing. Flow-Tools [11] is a powerful combination of a network flow collector and a flow analyzer. The flow collector can support single, distributed, and multiple servers for several NetFlow versions. The flow-capture module is used to collect NetFlow records (only User Datagram Protocol, no Stream Control Transmission Protocol) from network devices and store them in files in raw format. Then, either flow-print or flow-cat decodes the files for analyzing purposes. fdump [10] is a package providing tools to collect, process, and store network flow data. The nfdump toolkit also provides a converter for transforming flow-tools NetFlow records into nfdump formatted files. Nfdump stores flow records in timesliced files with a timestamp attached to the filename. Afterwards, particular files can be either merged or divided again in order to perform further analysis on it. Argus [1] is a real-time flow monitor, which is able to track and report network transactions it observes from packets collected on a network interface in promiscuous mode. The major difference to Cisco NetFlow records is that Argus flow records consider each traffic flow as a bidirectional sequence of packets that typically contain two sub-flows, one for each direction. There are two types of Argus records: the Management Audit Record and the Flow Activity Record, where the former provides information about Argus itself, and the latter provides information about network flows. DatCat [6] is a project that aimed at building a traffic traces repository including NetFlow data. Traces can be published by network operators using a web portal, while researchers can browse the existing repository and retrieve traces they need. One drawback of DatCat addressed in this paper is that all flow processing needs to be done on the client side. If the traffic trace is big, even if a researcher needs only a small part of the flow records (e.g., flow records with port 22 as a destination) the whole trace would have to be downloaded. 2.2

Flow Record Anonymization

Anonymization is the process of hiding the identity of a particular client of a network by inspecting IP packets sent. Anonymization of traffic traces cannot be applied for all applications. Some applications, e.g., accounting and traffic engineering, require the original traffic in order to assign costs to a particular user or to detect where and why a congestion appeared in the network. Several anonymization algorithms are known, but there is no standardized method for anonymization of traffic traces. The following anonymization techniques are encountered in different tools: • Truncation: This defines the basic type of IP address anonymization. The user chooses the number n of least significant bits to truncate from an IP address. E.g., truncating n=8 bits would replace an IP address with the corresponding class C network address.

4

C. Morariu, P. Racz, and B. Stiller

• Random permutations: With this method, a random permutation on the set of possible IP addresses is applied to translate IP addresses. A 32-bit block cipher represents a subset of permutations on the IP address space. The use of random hash tables implements a truly random permutation [9]. Thus, it is possible to generate any permutation during anonymization, not just one from a subset of the possible ones. • Prefix-preserving pseudonymization: This special class of permutations has a unique structure-preserving property: Two anonymized IP addresses match on a prefix of n bits, if and only if the unanonymized addresses match on a prefix of n bits as well. This is achieved by a user-supplied passphrase, which generates a cipher key that in turn determines the permutation to be applied. This allows for an anonymization with a consistent mapping between multiple anonymizers sharing a common key. Based on these mechanisms several anonymization tools are used in practice. TCPdpriv [12] implements a table-based, prefix-preserving translation of IP addresses. In order to maintain the consistency of the anonymization, it stores a set of binding pairs of IP addresses. Whenever a new IP address needs to be anonymized, it will be compared with all raw IP addresses within stored binding pairs for the longest prefix match. For remaining bits, where no prefix is stored inside the binding table, a pseudorandom function is applied. TCPdpriv removes sensitive information by operating only on packet headers The. TCP and UDP payload is simply removed, while the entire IP payload is discarded for protocols other than TCP or UDP. The tool provides multiple levels of anonymization, from leaving fields unchanged up to performing a more strict anonymization. CryptoPAn [14] is a cryptography-based network trace anonymization tool with a prefix-preserving property. In CryptoPAn the mapping from original IP addresses to anonymized IP addresses is one-to-one. To anonymize traces, CryptoPAn uses a secret key that is used across multiple traces. This allows multiple traces to be anonymized in a consistent way over time and across locations. That is, the same IP address in different traces is anonymized to the same address, even though traces might be anonymized separately at different time and/or at different locations. CryptoPAn is used in many different tools such as CANINE [9] or nfdump. AAPI (Anonymization Application Programming Interface) [8] is a C-based anonymization library. This tool provides a fast on-line anonymization support as well as composition, decomposition, and filtering of NetFlow version 5 and 9 traces previously saved in the tcpdump [13] format. AAPI uses randomizing, replacing, and prefixpreserving anonymization techniques. CAIE [9] addresses several aspects. It supports different anonymization methods on different fields of a flow record; IP address truncation, random permutations, and prefix-preserving pseudonymization; bilateral classification, black marker anonymization of port numbers; protocol number replacement; timestamp annihilation, random timeshift, timestamp enumeration, and byte count replacement. CANINE also acts as a powerful converter between different versions of NetFlow records.

Design and Implementation of Distributed Platform for Sharing IP Flow Records

2.3

5

Comparison and Discussion

TCPdpriv shows two major drawbacks: it only provides online anonymization and it cannot anonymize parallel traces within a distributed environment in a consistent way. CryptoPAn does not have these restrictions and it allows for a consistent anonymization between several anonymizers based on the shared cipher key. Furthermore, it is fast to anonymize flow records online. AAPI and CANINE provide more than IP address anonymization and allow hiding other IP fields as well. Besides several techniques for veiling different flow record fields, both support cryptographybased, prefix-preserving IP anonymization. In case operators decide to anonymize their data, researchers do not have a tool at their disposal to easily find these traces in the Internet, thus, they often have to spend time to look for new public traces. Often researchers use well known traffic traces over and over, and often algorithms which could be used to detect certain network problems (such as worms) could only be evaluated based on real traffic traces long after the problem appeared. DipSIF addresses these issues by providing a platform with which any NetFlow data owner can share data, while still being able to apply authorization policies and anonymize data before being sent. DipSIF integrates the widely used NetFlow data storage tools into existing repositories.

3

Design

In order to enable such a sharing between different institutions, the design of DipSIF focused on an architecture with access control and online traffic anonymization. Based on requirements determined, the resulting architecture is fully decentralized. 3.1

Requirements

Based on the investigation of distributed trace handling the following key requirements have been identified for DipSIF: • Retrieval of remotely stored NetFlow data: The major requirement is to supply a researcher with an Application Programming Interface (API) for querying flow records from any remote NetFlow repository of participating parties. • Access control: The platform shall offer an authentication mechanism and fine granular access control to NetFlow data stored. Each storage provider shall be in the position to decide independently, which users or groups of users will have access to flow traces collected. • Online anonymization: Any request for NetFlow records shall result in anonymized records being sent to the requester. The anonymization shall be done online and at least mutate IP addresses stored within NetFlow records in a prefix-preserving manner. • Encryption: Communication channels between all distributed components are typically not trustworthy, all communication shall be cryptographically secured.

6

3.2

C. Morariu, P. Racz, and B. Stiller

Architecture

Exploiting those requirements a three-tiered architecture (cf. Fig. 1) has been designed. The architecture consists of a retrieval service running on separate storage nodes (1st tier), an authentication and forwarding service on a set of server nodes (2nd tier), and a client library that offers an API (3rd tier) for accessing arbitrary NetFlow datasets from any of those repositories available. The secure communication is achieved by using Secure Socket Layer (SSL), offering client and server authentication, data integrity, and a cryptographically secured channel for communication between client and server. A query including optional filters and other control messages from clients results in anonymized NetFlow records being returned to the requester.

Fig. 1. Three tier architecture

While the first (and third) tier clearly show a server (and client) functionality, the second tier features a dual behavior: it acts as a client, when requesting NetFlow records from the local storage component, and at the same time it acts as a server, when handling requests received from remote clients. Fig. 2 illustrates the architectural design of the sharing platform. It can be seen in this figure how information is exchanged between a client, a server, and a storage node. The figure also shows how the different components are deployed and how they interact. The end-user using an API can build applications for NetFlow record retrieval according to his needs. The API provides a user with several methods that are used to control the dataset that will be requested (e.g., filters to be applied to reduce the dataset, transmission rate, etc.). In order to access any functionality of the API the client needs to pass an authentication process beforehand. After a successful authentication the client is enabled to request flow data. In a first step a request message containing the type of the desired dataset needs to be created, which is passed to the server by invoking an appropriate remote procedure defined in the API. When the server receives the client's request message, it checks the preconfigured authorization and anonymization policy using the Access Control module. The authorization result is logged and if the user is allowed to have access to the NetFlow repository the request is forwarded together with the anonymization policy to the storage component. The storage service searches the appropriate dataset in the repository, anonymizes the dataset according to the received anonymization policy, and finally applies desired filters, which are passed within the request message. The resulting dataset is returned by the replaying module (in the prototype implemented by nfdump tools) directly to the client as a stream of NetFlow packets.

Design and Implementation of Distributed Platform for Sharing IP Flow Records

7

Fig. 2. Architecture

3.3

Client Component

The client component serves as an entry point to DipSIF. It includes a library and its API, which can be used to communicate with other components. The library enables a user — possessing a valid certificate — to request flow traces collected from remote storage repositories. Requests of NetFlow records can be made using the API, which hides the actual retrieval process. Users shall only care about what (i.e. filters) data and from where (i.e. server URL) they require data, but not how it is obtained.

Fig. 3. A controllable capturing unit

The client library can buffer NetFlow records sent by the storage node's replay component and simultaneously deliver them to the application, built on top of the client library, at a controlled rate. Therefore, incoming NetFlow records are received and decoded by the Capture Thread that adds them to a temporary FIFO list, as illustrated in Fig. 3. Simultaneously, the Controller Thread pops out buffered NetFlow records according to user specified events. In addition to this mechanism, the request message includes a delay attribute for the purpose of adjusting the rate of NetFlow records sent by the storage component. This helps avoiding buffer overflows on the receiver side. Before sending a request to a selected server, the client application binds a receiver to an unused local port and attaches this information to the request message. The port number is used to tell the replaying module of the storage component, where to return the requested data to.

8

C. Morariu, P. Racz, and B. Stiller

The request message includes the following attributes: • User identity (i.e. accepted and valid certificate) • Desired server (e.g., emanicslab1.csg.uzh.ch) • Return address (i.e. IP:port) • Requested data (i.e. time-period) • Optional filters (e.g., port- and/or protocol-based) • Delay option (e.g., delay each record by 10 microseconds) Upon the receipt of a request message the server component replies with the result of the request authorization. 3.4

Server Component

The server component serves as a gateway for the client-to-storage communication. It performs client authentication and authorization and controls the access to storage repositories. Each server component maintains its own credential repository (truststore) that manages a list of X.509 certificates of users to which access is granted. Requesters need to authenticate by using their corresponding private key stored in their keystore before their query is forwarded to the storage node. A Public Key Infrastructure (PKI) additionally shares public keys in order to allow for an encryption of the communication channel using SSL. All servers use a predefined port for accepting incoming queries. The server interface allows client applications to communicate with server instances by calling remote methods defined by the interface using Remote Method Invocation (RMI). The server interface allows for retrieving the server's actual state and for sending request messages to storage components. Users with a valid and authorized certificate can request to order and filter the resulting dataset by means of the following options, or a combination thereof: • Time (i.e. start-/end-time of flow records) • Port numbers (e.g., traffic with destination port larger 80) • Protocol types (e.g., TCP — Transport Control Protocol, UDP — User Datagram Protocol, ICMP — Internet Control Messaging Protocol) In order to provide access control, every client interacting with the server needs to authenticate by means of a certificate. Only authenticated users will receive necessary privileges to retrieve flow records of any of the remote storage repositories. Server administrators can review a client’s certificate and decide, whether the certificate is added to the server’s truststore (cf. Fig. 4). The certificate can be issued by a local certification authority as well. The prototype implementation of the server supports two groups of users, administrators and certified clients. The first group, which includes administrators, is allowed to maintain server and storage components, grants registration requests from new users by accepting their certificates to the server's truststore, manages user accounts, and assigns anonymization settings to individual users and groups. The second group includes the clients to which access to NetFlow data is allowed. Certified

Design and Implementation of Distributed Platform for Sharing IP Flow Records

9

Fig. 4. Registration process

clients must have a valid and accepted certificate and can request flow data. Such requests can be made by applications residing on the client using an API that is described in Sect. 4. Clients are not authorized to modify any flow data and to request any unanonymized flow data. As the server component acts as an intermediary to all flow requests, additional authorization mechanisms could be envisioned, such as: access for a given group of users only to a subset of flow records (for example only to flow records belonging to a subnet), or access to flow records generated within a specific time frame. For the purpose of monitoring the platform's performance and controlling the data exchange, any incoming request and outgoing data stream is logged to a file by the server at the moment of delivery. The information included in the log file consists of the request timestamp, the requestor’s identity (username, usergroup, and IP address), anonymization options, the time period of requested flow records, filter types applied, and the processing time. 3.5

Storage Component

The storage component is responsible for the storage of NetFlow records. The DipSIF architecture does not define the way on how flow records are collected — any tool can be used to fill the storage component with flow records. NetFlow records can be stored in databases or directly in binary files in their raw format. Thus, the design of DipSIF is independent of the storage type. The storage interface of the storage component offers methods allowing for the searching and retrieving sets of flow records from a repository independent of the flow record's type and independent of the storage system (database or a raw files). The interface is capable of handling multiple heterogeneous storage repositories attached to the server component as illustrated in Fig. 5. Due to the fact that NetFlow

10

C. Morariu, P. Racz, and B. Stiller

Fig. 5. Multiple heterogeneous storage repositories attached to a server component

records come in many different, incompatible formats, the storage platform allows for handling more than one specific type of flow record. In this case the storage component translates all flow records into the NetFlow version 5 format. The decision of using NetFlow version 5 for transferring NetFlow records from remote repositories to clients that requested those records is due to the fact that most of the investigated repositories, use NetFlow version 5. Changing to NetFlow version 9, or IPFIX requires little change in the replaying unit of the storage component. For anonymizing IP packets, the CryptoPan [14] prefix-preserving IP address anonymization algorithm is applied on a dataset before being forwarded to a requester. The key for anonymization is created upon the authentication process by the server component and sent to the client once the authentication is successful. The flow data is stored in its original state and is not being anonymized until it leaves the storage node. During anonymization, only IP addresses are anonymized while other NetFlow fields are kept unchanged. The reason why those fields are kept unanonymized is that for many research approaches realistic network traces are needed. In order to increase the privacy protection, the platform rebuilds the cipher key that is used as a base for the anonymization procedure. For every incoming request, the server generates a session key that will be used to anonymize NetFlow records within that session. For higher flexibility, and more control on the NetFlow provider side, a future version of the prototype could use more anonymization options (such as anonymization of port numbers) which could be used depending on the application requirements for which the flow records will be used (e.g., by default the port numbers would be anonymized, but if the application requires that the port numbers are kept in their original form, a special authorization attribute could allow that).

4

Implementation

Based on the design above a prototype was implemented in Java using the Java Development Kit 5.0 (JDK5). An outline of the implementation architecture is shown in Fig. 6. As it can be seen in the figure, communication between the client and the server as well as between the server and the storage component uses Remote Method Invocation (RMI) calls. All communication between the client component and the server uses an SSL connection. The backward communication between the storage component and the client uses the NetFlow version 5 protocol. The prototype uses the

12

C. Morariu, P. Racz, and B. Stiller

The prototype includes storage modules for nfDump and FlowTools. In order to support other NetFlow storage formats the StorageAdapter interface needs to be implemented and specialized for those specific formats. The Shell class allows for the execution of Linux shell commands, a requirement for some storage toolkits that do not offer an API to access flow data, but only binary programs for handling that data. The shell commands are predefined in the storage component and a user cannot execute any shell commands directly.

5

Evaluation

Based on the introduced and important functional requirements of the flow record sharing platform (cf. Sect. 3.1), the design was done (cf. Sect. 3.2 to Sect. 3.5), and the implementation (Sect. 4) was undertaken. Thus, a functional evaluation is possible, while the running prototype shows the feasibility: • Retrieval of stored etFlow data: The DipSIF offers an API allowing for the query of NetFlow records from remote repositories. The library offers several instruments to control the kind of requested data as well as the received data stream. A set of available filter options allows for sorting requested NetFlow records by specific attributes, such as IP protocol or port numbers. The stream received is controlled by delaying individual NetFlow records, by limiting the amount of data, and by interrupting it for a given time period. • Access control: Authentication is implemented using a PKI while authorization uses access lists. Each storage provider is in the position to decide independently to which users access to collected flow traces is granted. Any client needs to hand over a signed certificate to the administrator in charge of the desired repository in order to be able to access the corresponding NetFlow data. • Online anonymization: Any NetFlow record is anonymized before being sent to the requester at the moment it leaves the storage node. The prototype implemented uses the CryptoPAn library to anonymize NetFlow records. The anonymization is done in a prefix-preserving manner. • Encryption: The communication between the client and server component is encrypted using an SSL connection. However, the NetFlow transfer between the storage component and the client is left unprotected but IPSec might be used to protect the record transfer. Another way to improve security during the transmission of NetFlow records would be the use of SCTP (Stream Control Transmission Protocol) and a transport layer security protocol ontop of it. The modular design as well as a storage interfaces which is independent of the storage technology allow implementation of add-on modules for different storage formats besides the nfdump format which was used in the prototype. The performance of the DipSIF platform is given by the performance of its individual components. As server and storage components may serve multiple requests at the same time, these components could become bottlenecks when a large number of requests come in.While the server component only has to perform minimal process-

Design and Implementation of Distributed Platform for Sharing IP Flow Records

13

ing for each incoming request (verify the user credentials, check its authorization policies) and thus is less probable of becoming overloaded, the storage component may become overloaded when a large number of requests need to be handled at the same time. The current prototype handles each incoming request independently of other, so when multiple requests arrive at the storage component, each of these requests triggers a lookup in the nfdump files which increases the disk access time.

6

Summary and Future Work

Network traces determine one of the wider testing data for researchers working in the field of traffic analysis. Access to such traces is often a problem, either because of administrative reasons, which prevent network operators to share their traffic data, or either because of the lack of a search tool for existing traffic traces in the Internet. The work presented here proposed the DipSIF architecture as a sharing platform for NetFlow records. On one hand, it allows owners of traffic traces to share their data with researchers and provides them with access control mechanisms to control who will have access to those traces. It also provides anonymization capabilities so that no IP address involved in the real traffic is revealed. On the other hand, it allows researchers and protocol designers with an easy and standard access to repositories residing remotely and provides tools for controlling the data that is actually retrieved. Such a control will help to decrease data transfers, because researchers can specify, which data from a dataset is important for their research and only download these data instead of downloading a whole traffic trace, which includes — besides the interested traffic data — those data, which are unnecessary for the current research question tackled. The prototype presented can be improved further by adding additional functionality. For example the authorization process can be extended in order to allow an administrator to limit explicit flow records to which an external user has access (e.g., external access could be allowed only to web traffic). Another useful extension is the possibility to perform processing on the remote storage in order to return just an aggregated value, rather than a large number of flow records. The UDP protocol used in the current implementation to transfer NetFlow records could be replaced with the SCTP protocol, which would add more security, reliability, congestion control awareness. As some traffic analysis tools use NetFlow records stored in files as input data, an option to transfer NetFlow records from the storage repository back to the requesting clients over HTTPS could be implemented.

Acknowledgment This work has been performed partially in the framework of the EU IST NoE EMANICS (FP6-2004-IST-026854). The authors are indebted to Nicolas Baumgardt, who implemented the prototype in his Student Thesis [2]. They also want to thank the reviewers who provided helpful feedback during the paper review process.

14

C. Morariu, P. Racz, and B. Stiller

References 1. Argus Homepage, http://www.qosient.com/argus/ (last access, April 2009) 2. Baumgardt, N.: Design and Setup of a Distributed Storage Repository for NetFlow Records; Student Thesis. CSG@IFI, University of Zürich, Switzerland (March 2008) 3. cflowd Homepage, http://www.sdsc.edu/~woodka/cflowd.html (last access April 2009) 4. Cisco NetFlow Homepage: http://www.cisco.com/en/US/products/ps6601/ products_iosprotocol_group_home.html (last access April 2009) 5. Claise, B. (ed.): Specification of the IP Flow Information Export (IPFIX) Protocol for the Exchange of IP Traffic Flow Information. IETF RFC 5101 (January 2008) 6. DatCat, Intenet Measurement Data Catalog, http://www.datcat.org/m (last access August 2009) 7. EMANICS Project Homepage, http://www.emanics.org/ (last access May 2009) 8. Koukis, D., Antonatos, S., Antoniades, D., Trimintzios, P., Markatos, E.P.: A Generic Anonymization Framework for Network Traffic. In: IEEE International Conference on Communications (ICC 2006), Istanbul, Turkey (June 2006) 9. Li, Y., Slagell, A., Luo, K., Yurcik, W.: CANINE: A Combined Conversion and Anonymization Tool for Processing NetFlows for Security. In: International Conference on Telecommunication Systems, Modeling and Analysis, Dallas, Texas, USA (November 2005) 10. nfdump Homepage, http://nfdump.sourceforge.net/ (last access April 2009) 11. Plonka, D.: FlowScan: A Network Traffic Flow Reporting and Visualization Tool. In: 14th USENIX Conference on System Administration, New Orleans, Louisiana, USA, December 2000, pp. 305–318 (2000) 12. TCPDpriv Homepage, http://ita.ee.lbl.gov/html/contrib/tcpdpriv.html (last access April 2009) 13. Tcpdump Homepage, http://www.tcpdump.org/ (last access May 2009) 14. Xu, J., Fan, J., Ammar, M., Moon, S.B.: On the Design and Performance of Prefixpreserving IP Traffic Trace Anonymization. In: 1st ACM SIGCOMM Workshop on Internet Measurement (IMW 2001), San Francisco, California, USA, November 2001, pp. 263–266 (2001)

Design of a Stream-Based IP Flow Record Query Language Vladislav Marinov and J¨ urgen Sch¨ onw¨ alder Computer Science, Jacobs University Bremen, Germany {v.marinov,j.schoenwaelder}@jacobs-university.de

Abstract. Analyzing Internet traﬃc has become an important and challenging task. NetFlow/IPFIX ﬂow records are widely used to provide a summary of the Internet traﬃc carried on a link or forwarded by a router. Several tools exist to ﬁlter or to search for speciﬁc ﬂows in a collection of ﬂow records, however the ﬁltering or query languages that these tools use have limited capabilities when it comes to describing more complex network activity. This paper proposes a framework and a new streambased ﬂow record query language, which allows certain types of traﬃc patterns to be deﬁned and matched in a collection of ﬂow records. The usage of the proposed new language is exempliﬁed by constructing a query identifying the Blaster.A worm. Keywords: Network measurement, NetFlow, IPFIX.

1

Introduction

The NetFlow protocol [1], originally designed by Cisco Systems, enables routers to export summary information about the traﬃc ﬂows that traverse a router. Inspired by Cisco’s early work, the IETF created a standard IP ﬂow information export protocol called IPFIX [2] A network ﬂow is deﬁned as an unidirectional sequence of packets between given source and destination endpoints. Speciﬁcally, a ﬂow is usually identiﬁed by the combination of the following seven key ﬁelds: source and destination IP address, source and destination port number, IP protocol type (TCP, UDP, etc.), ToS byte, and the input interface (ifIndex). In addition to the key ﬁelds, a ﬂow record contains other accounting ﬁelds such as packet and byte counts, input and output interfaces, bit-wise logical or of TCP ﬂags, timestamps, MPLS labels etc. Network elements (routers and switches) gather ﬂow data and export it to collectors for analysis. The ﬂow records exported via NetFlow/IPFIX provide a summary about the traﬃc traversing a speciﬁc router. However, raw collections of ﬂow records still contain too many details for network administrators and they are not useful unless processed by network analysis tools. Most of the existent ﬂow record processing tools provide mechanisms for searching of speciﬁc ﬂows through some simple operations like ﬁltering by an IP address or port number or generating Top-N talkers reports. However, in order to match more complex ﬂow patterns against collections of ﬂow records, one needs a useful ﬂow record query language. C. Bartolini and L.P. Gaspary (Eds.): DSOM 2009, LNCS 5841, pp. 15–28, 2009. c IFIP International Federation for Information Processing 2009

16

V. Marinov and J. Sch¨ onw¨ alder

Given the large number of ﬂow records collected on high-speed networks, it is necessary to reduce their number to a comprehensible scale using ﬁltering and aggregation mechanisms. Each ﬂow or aggregated ﬂow has a set of properties attached to it that characterize the ﬂow. It is to be expected that ﬂows that correspond to similar network activities (certain applications or certain attacks) have similar properties. In addition to the properties recorded in ﬂow records, one can derive further properties that are even more suitable to characterize the behavior of a ﬂows. One objective when investigating traces is to detect traﬃc regularities such as repeating patterns, which can be associated with the usage of common network services. This approach can be further extended to detect traﬃc irregularities such as network anomalies or attacks, which also generate speciﬁc patterns. These patterns typically spread over several ﬂows. For example, if an intensity peak in ﬂow X always occurs after an intensity peak in ﬂow Y with a ﬁxed delay, they form a pattern describing a certain network behavior. The goal of network administrators is to detect such patterns of correlated ﬂows. For example, one would be interested in ﬁnding out where, when, and how often a certain Internet service is used. A concrete scenario is a network administrator who wants to detect VoIP applications by ﬁnding STUN ﬂows generated by VoIP applications when they try to discover whether they are located behind a Network Address Translator (NAT). If one knew the pattern that is created when a service is trying to establish a connection, one could search for this speciﬁc pattern in the selected ﬂows. We are aware that although the presence/absence of a certain pattern may be a hint for the presence/absence of a particular service this by no means proves that the service is really running/missing. In this paper we propose a ﬂow record query language, which allows to describe patterns in a declarative and easy to understand way. This paper is a followup of the early paper [5], where we discussed in some detail the motivation of this research. The proposed language is able to deﬁne ﬁlter expressions (needed to select relevant ﬂows) and relationships (needed to relate selected ﬂows). It allows to express causal dependencies between ﬂows as well as timing and concurrency constraints. Existent query languages as discussed in Section 2 are not suitable for detecting complex traﬃc patterns because of either performance issues (SQL-based query languages) [3,4] or because they lack a time and concurrency dimension (BPF expressions and the other query languages we discuss). Furthermore, the new query language provides support for network speciﬁc aggregation functions, such as IP address preﬁx aggregation, IP address suﬃx aggregation, port number range aggregations, etc. which are not part of many query languages. Using the new query language, we built a knowledge base of ﬂow ﬁngerprints that belong to some common network services, applications and attacks. As an example, we describe the query detecting the ﬂow ﬁngerprint of the Blaster.A worm. The rest of the paper is structured as follows. Section 2 provides a short survey of existing ﬂow ﬁltering and query languages. In Section 3 we present our stream-based ﬂow query language and in Section 4 we show an application

Design of a Stream-Based IP Flow Record Query Language

17

example by using it to describe a common network traﬃc pattern. We conclude in Section 5 with a few remarks on ongoing work.

2

Related Work

According to [5] existing ﬂow record query languages can be split into SQL-based query languages, ﬁltering languages, and procedural languages. 2.1

SQL-Based Query Languages

Many of the early implementations of network analysis tools used a Relational Database Management System (RDBMS) to store the data contained in ﬂow records and therefore they use SQL-based query languages for retrieving ﬂows. B. Nickless [6] describes a system which uses standard MySQL and Oracle DBMS for storing the attributes of NetFlow records. Using powerful SQL queries, the tool was able to provide good support for basic intrusion detection and usage statistics. With the advance of high-speed links, however, network managers could not rely on pure DBMS anymore because of performance issues. There was also a semantic mismatch between the traﬃc analysis operations and the operations supported by the commercial DBMS. The data used by network analysis applications can be best modeled as transient data streams as opposed to the persistent relational data model used by traditional DBMS. It is recognized that continuous queries, approximation and adaptivity are some key features that are common for such stream applications. However, none of these is supported by standard relational DBMS. Based on these requirements B. Babcock et al. [4] propose the design of a Data Stream Management System (DSMS). Together with the model the authors also extend the SQL query language so that the DSMS can be queried over time and provide examples of network trafﬁc reports that are generated based on ﬂow data that is stored in such a DSMS. Gigascope [7] is another stream database for network monitoring applications. It uses GSQL for query and ﬁltering, which is yet another modiﬁcation of the SQL query language adopted in a way so that time windows can be deﬁned inside the query. Tribeca [3] is another extensible, stream-oriented DBMS designed to support network traﬃc analysis. It is optimized to analyze streams coming from the network in real time as well as oﬄine traces. It deﬁnes its own stream-based query language which supports operations such as projection, selection, aggregation, multiplexing and demultiplexing of streams based on stream attributes. The query language also deﬁnes a windowing mechanism to select a timeframe for the analysis. 2.2

Filtering Languages

The Berkeley Packet Filter (BPF) [8] allows users to construct simple expressions for ﬁltering network traces by IP address, port number, protocol etc. and translates these expressions into small programs executed by a generic packet

18

V. Marinov and J. Sch¨ onw¨ alder

ﬁltering engine. One popular use of the BPF is in the tcpdump utility. The BPF rules for constructing ﬁlter expressions are also used in nfdump [9], which is a powerful and fast ﬁlter engine used to analyze network ﬂow records. nfdump is currently one of the de facto standard tools for analyzing NetFlow data and generating reports. BPF expressions are also used in the CoralReef network analysis tool described in [10,11] in order to generate traﬃc reports from collected trace ﬁles. The Time Machine tool described in [12] uses BPF expressions to deﬁne classes of traﬃc and BPF is also part of the query language used by the tool for retrieval of interesting traﬃc. The flow-tools package [13] is another widely-used collection of applications for collecting and analyzing NetFlow data. Two of the ﬂow-tools applications are responsible for ﬁltering ﬂows and generating reports: flow-filter and flow-report. The former application uses the Cisco Access Control List (ACL) format to specify a ﬁlter for IP addresses and command line arguments for specifying other ﬁltering parameters such as port numbers, ASes etc. The latter uses a conﬁguration ﬁle where reports can be deﬁned by using a number of primitives. 2.3

Procedural Languages

FlowScan described in [14] is a collection of perl scripts which glues together a ﬂow-collection engine such as the flow-capture application from flow-tools, a high performance RRD database, which is speciﬁcally designed for time series data [15], and a visualization tool. FlowScan has the capability of generating powerful high-level traﬃc reports, which might help operators to detect interesting traﬃc patterns. However, reports must be speciﬁed as separate perl modules, which is not trivial and might involve some heavy scripting. C.Estan et al. [16] proposes an approach for detecting high-level traﬃc patterns by aggregating NetFlow records in clusters based on ﬂow record attributes. Aggregation on several ﬂow attributes results in a multidimensional cluster. Initially all possible multidimensional clusters are constructed. Then an algorithm is executed which selects only clusters that are interesting to the network administrator. It aims at retaining clusters with the least degree of aggregation (so that a bigger number of ﬂow attributes is contained). Interesting activities are considered to be exceeding a certain threshold of traﬃc volume of a cluster or signiﬁcant change of the traﬃc volume inside the cluster. Finally, all clusters are prioritized by being tagged with a degree of unexpectedness and presented to the network administrator as a traﬃc report. The SiLK Analysis Suite [17] is another script-based collection of commandline tools for querying NetFlow data. It provides its own primitives for deﬁning ﬁltering expressions. Unlike other network analysis tools, SiLK contains two applications that allow an analyst to label a set of ﬂows sharing common attributes with an identiﬁer. The rwgroup tool walks through a ﬁle of ﬂow records and groups records that have common attributes, such as source/destination IP pairs. This tool allows an analyst to group together all ﬂows in a long lived session such as a FTP connection. rwmatch creates matched groups, which consist

Design of a Stream-Based IP Flow Record Query Language

19

of an initial record (a query) followed by one or more responses. Its most basic use is to group records into both sides of a bidirectional session, such as a HTTP request.

3

Stream-Based Flow Query Language

Our framework for IP ﬂow ﬁltering follows a stream-oriented approach — it consists of a number of processing elements or operators, which are connected with each other via pipes. Each element receives an input stream, performs some sort of operation on it (ﬁltering, aggregation etc.) and the output stream is piped to the next element. Figure 1 shows the framework and in the following sections we describe each of its elements. A complete deﬁnition of the syntax and the semantics of the elements can be found in [19]. Section 4 provides an example illustrating the usage of the primitives of the stream-based ﬂow query language. The names of the ﬁltering primitives in our language are closely linked to the ﬂow record attributes in RFC 5102 [18].

Fig. 1. IP ﬂow ﬁltering framework

3.1

Splitter

The splitter is the simplest operator in the IP ﬂow ﬁltering framework. It takes the input stream of ﬂow records and copies them on each output stream without performing any changes on them. There is one input branch and several output branches for a splitter. 3.2

Filter

The filter operator takes a stream of ﬂow records as input and copies to its output stream only the ﬂow records that match the ﬁltering rules. The ﬂow records, which do not match the ﬁltering rules are dropped. The filter operator performs absolute ﬁltering, it compares the ﬂow attributes of the input

20

V. Marinov and J. Sch¨ onw¨ alder

ﬂow records with absolute values (or a range of absolute values). It can also perform comparison between the various ﬁelds of a single ﬂow record, that is it can compare one ﬁeld of a ﬂow record against another ﬁeld of the same ﬂow record (for example source port number with destination port number). The filter operator does not support relative ﬁltering between ﬁelds from diﬀerent ﬂow records i.e., it does not perform comparison between the ﬂow attributes of diﬀerent incoming ﬂow records. 3.3

Grouper

The grouper operator takes a stream of ﬂow records as input and partitions them into groups and subgroups following grouping rules. The grouping rules themselves are organized into rule modules, where each rule module contains a number of rules logically linked by an implicit logical and. The diﬀerent rule modules on the other hand are logically linked by an implicit logical or. The rules reﬂect some relative dependencies and patterns among the attributes of the input ﬂow records. The grouper tags each ﬂow record with a group label and each group consists of ﬂow records tagged with the same group label. Internally, each group consists of several not necessarily non-overlapping subgroups, which correspond to the diﬀerent rule modules. The grouper also tags each ﬂow record with a rule module identiﬁer (also called a subgroup label) if the ﬂow record satisﬁes the set of rules within the corresponding rule module. In order to be added to a group a ﬂow record must satisfy the rules from at least one rule module. In case a ﬂow record satisﬁes the rules from several rule modules, it is tagged with the rule module identiﬁer of all matching rule modules and thus belongs to several subgroups. The way group and subgroup labels are stored into ﬂow records is implementation speciﬁc, for example the SiLK tool [17] stores the labels in the next-hop ﬁeld of the ﬂow records. For each group of ﬂow records a group record is created. It may consists of the following attributes: – Flow record attributes according to which the grouping was performed. This is usually a set of attributes that are unique for a subgroup and form a key for that subgroup. If there is a single rule module within a grouper deﬁnition then there will be a single subgroup and these ﬂow record attributes will be unique for the group as well. – Aggregated values for other ﬂow record attributes from a single subgroup. If the subgroups within a single group are identiﬁed by g1, g2 etc. then the group record may contain members such as g1.sum(attr1), g1.min(attr2), g2.max(attr3) etc. – Aggregated values for other ﬂow record attributes from the whole group. In such a case the aggregation is performed over the ﬂow records of the whole group (as opposed to aggregation over a single subgroup). For example, the group record may contain members such as sum(attr1), min(attr2), max(attr3) etc. Note that in this case we can drop the subgroup label. Once the group record is created the subgroup labels are not needed anymore and can be deleted. Finally, the group records are copied over the output stream.

Design of a Stream-Based IP Flow Record Query Language

21

During the grouping operation some information from the original ﬂow record trace is lost because of the aggregation operation during the creation of the group records. Therefore, after the grouping and tagging is performed and before the aggregation phase, the tagged ﬂow records are copied to a temporary storage so that they can be later retrieved by the ungroup operator. The details of the algorithm are described in [19]. 3.4

Group-Filter

The group-filter operator takes a stream of group records as input and copies to its output stream only those group records that match the ﬁltering rules. The group records, which do not match the ﬁltering rules are dropped. The group-filter operator performs absolute ﬁltering on the ﬂow record attributes or the aggregated ﬂow record attributes contained within the group records. It compares the ﬂow record attributes (aggregated ﬂow record attributes) of the input group records with absolute values (or a range of absolute values). It can also compare various group record attributes within the same group record. It does not support relative ﬁltering i.e., it does not perform comparison between the ﬂow record attributes (aggregated ﬂow record attributes) of diﬀerent incoming group records. 3.5

Merger

The merger operator takes several streams of group records as input and copies to its output tuples of group records that satisfy some pre-deﬁned merging rules. The merging rules are organized in merging rule modules. Each rule module speciﬁes a set of input branches from all branches that meet at the merger and a number of rules, which use group record attributes to deﬁne certain relationships among the various ﬂow groups. If there are N input branches as input to a speciﬁc merging rule module, the output stream of that rule module will consist of N tuples of group records, one group record per input branch. The output stream of one of the merging rule modules is the output of the whole merger operator. There is always exactly one rule module that produces the output stream for the merger operator and that rule module must be the ﬁrst one deﬁned in the merger deﬁnition. In most cases there will be only one merging rule module and the tuples of group records that it generates will produce the merger output stream. Using one merging rule module allows us to deﬁne the existence of certain patterns among the various ﬂow groups. However, in order to be able to check for patterns that do not exist we will need more than one merging rule module. For example consider the following two queries: Q1 : Find the ﬂow records that correspond to a TCP connection between source IP address A and destination IP address B, port ftp, followed by a TCP connection between source IP address B, port ftp-data and destination IP address A.

22

V. Marinov and J. Sch¨ onw¨ alder

Q2 : Find the ﬂow records that correspond to a TCP connection between source IP address A and destination IP address B, port ftp, followed by a TCP connection between source IP address B, port ftp-data and destination IP address A only if these two connections are not preceded by a TCP connection between source IP address A and destination IP address B, port http. In more straightforward words, the ﬁrst query aims at detecting a FTP ﬁle transfer between A and B, while the second query aims at detecting a FTP ﬁle transfer between A and B only if A did not already download the respective ﬁle using HTTP. While query Q1 is relatively easy to deﬁne using a single rule module, for query Q2 we will need a more complicated mechanism to check for the condition “A HTTP transfer did not already take place between these two entities”. In such a scenario we ﬁrst perform the merging using the module rules that produce the merger output stream. Before copying the resulting tuples to the output stream, however, we feed them into the other merging modules and for each such tuple we check if the corresponding merging module would produce some output. These additional merging rule modules are used to deﬁne certain patterns that should not be observed and therefore a resulting tuple is only copied to the merger output stream if it does not generate any result when fed into any of the additional merging rules modules. In general, the merger operator allows to perform grouping at a more general level compared to the grouper operator. Another powerful feature of the merger operator is its capability to express timing and concurrency constraints among various traﬃc groups by using Allen’s time interval algebra [20]. 3.6

Ungrouper

The ungrouper operator takes a stream of group record tuples as an input. For each group record tuple it expands the ﬂow groups contained in the tuple using the labels and the ﬂows stored in the temporary storage during the grouping phase of the grouper operator. Finally, for each group record tuple it outputs a separate stream of ﬂow records. The ﬂow records obtained from each group record tuple are ordered by their timestamps and presented to the viewer in capture order. Any ﬂow record repetitions are eliminated, that is if a ﬂow record is part of several ﬂow groups within the group record tuple it is shown to the viewer only once. Each output stream is considered to be a result/match of the query operation performed by using the IP ﬂow ﬁltering framework. A query operation may return any number of results or no results at all and should clearly distinguish the ﬂow records that belong to diﬀerent results.

4

Application

In this section we present the traﬃc pattern generated by a computer infected with the Blaster.A worm and deﬁne this pattern using our stream-based ﬂow

Design of a Stream-Based IP Flow Record Query Language

(a) Packet level breakdown

23

(b) Flow level breakdown

Fig. 2. Packet level and ﬂow level breakdown of a Blaster infection

record query language. The Blaster.A worm [21] is a recent Internet worm, which exploits the Microsoft Windows Remote Procedure Call DCOM vulnerability. Upon successful execution, the worm attempts to retrieve a copy of the ﬁle msblast.exe from the compromising host. Once this ﬁle is retrieved, the compromised system then runs it and begins scanning for other vulnerable systems to compromise in the same manner. D¨ ubendorfer et al. [22] describe the various stages of the Blaster worm infection and perform an in-depth packet and ﬂow level analysis. The characteristic network activity (an infected attacker trying to infect other hosts on the network) associated with such an attack consists of the steps described in Figure 2(a). The ﬂow-level breakdown of the Blaster attack is shown in Figure 2(b). The ﬂow record ﬁngerprint of a Blaster infection consists of the following sequence of ﬂows (order is important): – More than 20 ﬂows originating from the attacker directed to port 135 of diﬀerent hosts. These ﬂows are small since they only carry a SYN packet. This represents the scanning activity of the attacker. Some of these scans may trigger a reverse ﬂow consisting of RST packets. – In a successful attack there will be a pair of bigger ﬂows (longer and carrying more data) to and from port 135/TCP of the victim – The pair of ﬂows representing the TCP connection to port 135 of the victim is followed by a pair of ﬂows representing the TCP connection to port 4444. – The next step is a pair of ﬂows to and from port 69/UDP of the attacker representing the TFTP transfer of msblast.exe. The ﬂow to the attacker slightly precedes the ﬂow from the attacker since the connection is initiated by the infected host. These two ﬂows end before the ﬂows representing the TCP connection on port 4444 from the previous step In order to describe a Blaster worm infection in our IP ﬂow ﬁltering framework we use the deﬁnitions below. We deﬁne one branch for each Blaster stage as speciﬁed in the ﬂow ﬁngerprint.

24

V. Marinov and J. Sch¨ onw¨ alder

The ﬁrst branch detects the scanning activity performed by the attacker. Initially, the f scan ﬁlter picks out the ﬂow records that have a destination port 135/TCP. Then the grouper g scan, which consists of a single group module g1, partitions the ﬂow records into groups, which have the same source IP address and the destination IP addresses consist of a block of subsequent IP addresses. splitter S{} filter f_scan { dstport = 135 proto = tcp } grouper g_scan { module g1 { srcip = srcip dstip = dstip relative-delta 1 stime = stime relative-delta 5ms stime = stime absolute-delta 5s } aggregate srcip, union(dstip), min(stime) as stime, max(etime) as etime, count } group-filter gf_scan { count > 20 } filter f_victim { srcport = 135 OR dstport = 135 proto = TCP } grouper g_group_tcp { module g1 { srcip = dstip dstip = srcip srcport = dstport dstport = srcport stime = stime relative-delta 5ms } module g2 { srcip = srcip dstip = dstip srcport = srcport dstport = dstport stime = stime relative-delta 5ms } aggregate g1.srcip as srcip, g1.dstip as dstip, min(stime) as stime, max(etime) as etime } filter f_control { srcport = 4444 OR dstport = 4444 proto = tcp }

filter f_tftp { srcport = 69 OR dstport = 69 proto = udp } grouper g_tftp { module g1 { srcip = dstip dstip = srcip srcport = dstport dstport = srcport stime = stime relative-delta 5ms } module g2 { srcip = srcip dstip = dstip srcport = srcport dstport = dstport stime = stime relative-delta 5ms } aggregate g1.srcip as srcip, g1.dstip as dstip, min(stime) as stime, max(etime) as etime, g2.sum(bytes) as bytes } group-filter gf_tftp { bytes > 6K } merger M { module m1 { branches A,B,C,D A.srcip = B.srcip A.srcip = C.srcip A.srcip = D.dstip B.dstip = C.dstip B.dstip = D.srcip B.dstip in union(A.dstip) A < B OR A m B OR A o B B o C D d C } export m1 } ungrouper U{} input -> S S branch A -> f_scan -> g_scan -> gf_scan -> M S branch B -> f_victim -> g_group_tcp -> M S branch C -> f_control -> g_group_tcp -> M S branch D -> f_tftp -> g_tftp -> gf_tftp -> M M -> U -> output

Additional constraints ensure that a scanning attack lasts at most 5s and each scanning attack should not be interrupted for more than 5ms (since attackers usually generate the small SYN packets at a very high rate). Finally, for each

Design of a Stream-Based IP Flow Record Query Language

25

Fig. 3. Capturing Blaster worm infections with the IP ﬂow ﬁltering framework

group the grouper operator retains the source IP address, the set of destination IP addresses, the start and end time of the attack as well as the number of ﬂows in the group. Each such ﬂow group now contains information about the scan attack performed by a single host. The newly created group records are passed to the group-ﬁlter operator, which ﬁlters only these group records that refer to ﬂow groups containing more than 20 ﬂows. That is, we consider the traﬃc from a ﬂow group a scanning attack only if the attacker has scanned more than 20 hosts. If the attacker has scanned less than 20 hosts we consider the ﬂow group normal traﬃc activity and drop it. The second branch, which consists of the ﬁlter f victim and the grouper g group tcp aims at capturing the TCP connection that the attacker established with the victim i.e., a TCP connection between the attacker and port 135 of the victim. The ﬁlter picks out ﬂows with a source or destination port 135. The grouper g group tcp then aggregates all ﬂow records that correspond to the same TCP connection in one group. The group module g2 adds to the ﬂow group all ﬂow records that have the same source and destination transport endpoints as the ﬂow record that generated the ﬂow group. The group module g1 adds the ﬂow records that correspond to the opposite direction of the TCP connection i.e., it adds those ﬂow records for which the destination transport endpoint is the same as the source transport endpoint of the ﬂow record that generated the group and vice versa (the source transport endpoint is the same as the destination transport endpoint of the ﬂow record that generated the group). For each TCP connection then g group tcp retains the source and destination IP addresses as well as the start and end times for each group. The third branch aims at identifying the control connection between the attacker and port 4444/TCP of the victim. The ﬁlter f control picks out the ﬂow records with a source or destination port number 4444/TCP and the grouper g group tcp partitions them into groups as already explained. The last branch aims at capturing the TFTP download, which gets initiated by the victim host. The ﬁlter f tftp picks out the ﬂow records, which belong to UDP traﬃc to or from port 69 (tftp). Then the grouper g tftp is very much like g group tcp in terms of the partitioning that it is performing. Both groupers contain two group modules, which aim at detecting the forward and backward direction of each TCP connection / UDP transfer. g group tcp partitions the

26

V. Marinov and J. Sch¨ onw¨ alder

incoming stream of TCP ﬂow records into groups so that each group corresponds to a separate TCP connection and g tftp partitions the incoming stream of UDP ﬂow records into groups so that each group corresponds to a separate TFTP/UDP transfer (to the extent to which we are able to distinguish diﬀerent UDP/TFTP transfers by using the delta and relative-delta parameters). In general the grouper is not aware of what the incoming stream of ﬂow records contains, thus it is not aware if it is grouping TCP or UDP ﬂow records. Therefore, g group tcp can already do the job of splitting the incoming ﬂow records into groups where each group represents a separate UDP/TFTP transfer. In this case however, we are also interested to retain the amount of data exchanged in each UDP/TFTP transfer in order to do some ﬁltering later in the group-filter. Therefore, the speciﬁcation and the interpretation of g tftp is the same as the one of g group tcp i.e., the group module g2 adds to the ﬂow group all ﬂow records that belong to the forward direction of the UDP/TFTP transfer (as compared to the ﬁrst ﬂow records that generated the group) and the group module g1 adds all ﬂow records that correspond to the backward direction of the UDP/TFTP transfer. The only addition of g tftp as compared to g group tcp is that the former also retains the amount of data exchanged within each group (that is within each TFTP transfer).The resulting group records are then passed to a group-ﬁlter which retains only these group records, which represent a TFTP exchange of at least 6K since the netﬂow ﬁngerprint of this stage speciﬁes that the virus is approx. 6176 bytes. The next step consists of deﬁning the merger M and the merging conditions. M contains a single merging rule module m1, which takes the four already deﬁned branches A, B, C and D as an input. The ﬁrst three rules from the merging rule module deﬁnition specify that the source IP address of the attacker should be the same during the diﬀerent stages of the Blaster infection (since we want to retrieve a single attack from a single attacker host to a single victim host). The next three rules specify that the IP address of the victim should also stay the same and that the victim should be one of the scanned hosts from the ﬁrst stage of the Blaster infection. The last three rules express the time and concurrency constraints among the four branches using Allen’s time interval algebra. We assume that the scanning phase takes place completely before, meets or overlaps with the stage of successful TCP connection establishment on port 135 with the victim. Furthermore, the connection on port 135/TCP overlaps with the control connection on port 4444/TCP. Finally from Figure 2(b) one notices that the TFTP transfer happens during the control connection (i.e., the TFTP transfer is entirely contained within the control connection). The last part of our deﬁnition consists of deﬁning the ungrouper and linking the already deﬁned elements using pipes to build a model for our IP ﬂow ﬁltering framework.

5

Conclusions

After a careful analysis of the pros and cons of the existing ﬁltering and query languages [5], we designed a new IP ﬂow ﬁltering framework and an associated

Design of a Stream-Based IP Flow Record Query Language

27

ﬁltering language. The language primitives were chosen in such a way that the new IP ﬂow record query language has the capability to describe aggregation and comparison based on ﬂow record attributes. In addition various dependencies such as ﬂow correlation, timing and concurrency constraints, ﬂow ordering and causal relationships can be expressed. The IP ﬂow ﬁltering framework has a limited number of operators, which can be deﬁned and linked in a very ﬂexible manner. This makes it relatively straightforward to use for the deﬁnition and detection of various traﬃc patterns. In order to evaluate our new IP ﬂow record query language, we collected a set of traﬃc patterns that belong to some popular network applications and services [19]. We analyzed HTTP and FTP transfers, the propagation of some well-known worms as well as Skype traﬃc. The ﬂow ﬁngerprint of each traﬃc pattern was derived and written down using the IP ﬂow record query language. We are currently implementing a prototype consisting of a parser, which reads the ﬂow pattern deﬁnition, and an execution engine, which implements the operators or our IP ﬂow ﬁltering framework. For some of the more complex operators such as the grouper and the merger, there will be a need to do some research in order to decide which algorithms and heuristics should be used in order to optimize the performance. In addition, one should consider various possibilities for ﬂow storage and choose the most eﬃcient one. In the future, we envision that protocol experts will, assisted by an interactive ﬂow visualization system, develop queries for speciﬁc scenarios. Once our implementation is complete, we can test these queries and share them with non-experts through a knowledge base so that they can be easily matched against ﬂow traces.

Acknowledgement The work reported in this paper is supported by the EC IST-EMANICS Network of Excellence (#26854).

References 1. Claise, B.: Cisco Systems NetFlow Services Export Version 9. RFC 3954, Cisco Systems (October 2004) 2. Claise, B.: Speciﬁcation of the IP Flow Information Export (IPFIX) Protocol for the Exchange of IP Traﬃc Flow Information. RFC 5101, Cisco Systems (January 2008) 3. Sullivan, M., Heybey, A.: Tribeca: a System for Managing Large Databases of Network Traﬃc. In: Proceedings of ATEC 1998, pp. 13–24. USENIX Association, Berkeley (1998) 4. Babcock, B., Babu, S., Datar, M., Motwani, R., Widom, J.: Models and issues in Data Stream Systems. In: Proceedings of PODS 2002, pp. 1–16. ACM, New York (2002) 5. Marinov, V., Sch¨ onw¨ alder, J.: Design of an IP Flow Record Query Language. In: Hausheer, D., Sch¨ onw¨ alder, J. (eds.) AIMS 2008. LNCS, vol. 5127, pp. 205–210. Springer, Heidelberg (2008)

28

V. Marinov and J. Sch¨ onw¨ alder

6. Nickless, B.: Combining Cisco NetFlow Exports with Relational Database Technology for Usage Statistics, Intrusion Detection, and Network Forensics. In: Proceedings of LISA 2000, pp. 285–290. USENIX Association, Berkeley (2000) 7. Cranor, C., Johnson, T., Spataschek, O., Shkapenyuk, V.: Gigascope: A Stream Database for Network Applications. In: Proceedings of SIGMOD 2003, pp. 647– 651. ACM, New York (2003) 8. McCanne, S., Jacobson, V.: The BSD Packet Filter: A New Architecture for Userlevel Packet Capture. In: Proceedings of USENIX 1993, pp. 259–270. USENIX Association, Berkeley (1993) 9. Haag, P.: nfdump, http://nfdump.sourceforge.net/ 10. Moore, D., Keys, K., Koga, R., Lagache, E., Claﬀy, K.: The Coral Reef Software Suite as a Tool for System and Network Administration. In: Proceedings of LISA 2001, pp. 133–144. USENIX Association, Berkeley (2001) 11. Keys, K., Moore, D., Koga, R., Lagache, E., Tesch, M., Claﬀy, K.: The Architecture of CoralReef: an Internet Traﬃc Monitoring Software Suite. In: Proceedings of PAM 2001, CAIDA, RIPE NCC (2001) 12. Kornexl, S., Paxson, V., Dreger, H., Feldmann, A., Sommer, R.: Building a Time Machine for Eﬃcient Recording and Retrieval of High-Volume Network Traﬃc. In: Proceedings of IMC 2005. USENIX Association, Berkeley (2005) 13. Fullmer, M.: ﬂow-tools, http://www.splintered.net/sw/flow-tools/ 14. Plonka, D.: FlowScan: A Network Traﬃc Flow Reporting and Visualization Tool. In: Proceedings of LISA 2000, pp. 305–318. USENIX Association, Berkeley (2000) 15. Oetiker, T.: RRDTool, http://oss.oetiker.ch/rrdtool/ 16. Estan, C., Savage, S., Varghese, G.: Automatically Inferring Patterns of Resource Consumption in Network Traﬃc. In: Proceedings of SIGCOMM 2003, pp. 137–148. ACM, New York (2003) 17. Collins, M., Kompanek, A., Shimeall, T.: Analysts’ Handbook: Using SiLK for Network Traﬃc Analysis. CERT. 0.10.3 edn. (November 2006) 18. Quittek, J., Bryant, S., Claise, B., Aitken, P., Meyer, J.: Information Model for IP Flow Information Export. RFC 5102, Cisco Systems (January 2008) 19. Marinov, V.: Design of an IP Flow Record Query Language. Master’s thesis, Jacobs University Bremen (May 2009) 20. Fin, A.: A Genetic Approach to Qualitative Temporal Reasoning with Constraints. In: Proceedings of ICCIMA 1999, Washington, DC, USA. IEEE Computer Society, Los Alamitos (1999) 21. Symantec: W32.Welchia.Worm (August 2003) 22. D¨ ubendorfer, T., Wagner, A., Hossmann, T., Plattner, B.: Flow-Level Traﬃc Analysis of the Blaster and Sobig Worm Outbreaks in an Internet Backbone. In: Julisch, K., Kr¨ ugel, C. (eds.) DIMVA 2005. LNCS, vol. 3548, pp. 103–122. Springer, Heidelberg (2005)

Monitoring Multiple Concurrent Service Level Parameters with Multidimensional Trees Andreas Kiefer1 , Elias P. Duarte Jr.1 , and Cristina D. Murta2 1

Dept. Informatics, Federal University of Paran´ a (UFPR) P.O. Box 19081 Curitiba PR 81.531-980 Brazil {andreas,elias}@inf.ufpr.br 2 Department of Computing, CEFET-MG Belo Horizonte 30.510-000 Brazil [email protected]

Abstract. The introduction of new computing paradigms in the Internet as well as the increasing size and complexity of services and resources demand the development of new approaches for deﬁning and monitoring service levels. It is often necessary to keep track of multiple concurrent service level requirements. In this paper we present a service level monitoring strategy that allows both online and oﬄine tracking the performance of multiple concurrent resources. Data is collected with SNMP (Simple Network Management Protocol). The strategy is based on building multidimensional search trees. k-d (k-dimensional) trees are employed for online continuous monitoring, and k-d-B trees are employed for oﬄine monitoring, based on logs of monitored data. Searching with the proposed strategy has cost O(logN ) where N is the number of samplings or log size. The strategy allows clients and providers to conﬁrm whether contract speciﬁcations were hold or not, and for how long. Experimental results are presented, including a comparison of the proposed approach with a traditional database. A practical tool was implemented and results are shown for a set of monitored Web and Video servers, as well as for monitoring data obtained from a real Telecom billing system.

1

Introduction

Cloud computing [1], as well as utility [2] and grid computing [3] have changed the requirements of Internet users. In these systems, services and resources are shared, being provided to diverse customers by several providers. Customers need the assurance of performance and dependability levels, typically speciﬁed by service contracts [4,5]. A service contract is a formal service agreement deﬁned with a Service Level Agreement - SLA [6]. A contract deﬁnes the common understanding about the service, obligations, responsibilities, priorities, guarantees, and also the minimum service level that is acceptable for the customer, as well as penalties in case the service levels are not met. These speciﬁcations usually involve several parameters and diﬀerent combinations of desired levels [4,5]. C. Bartolini and L.P. Gaspary (Eds.): DSOM 2009, LNCS 5841, pp. 29–42, 2009. c IFIP International Federation for Information Processing 2009

30

A. Kiefer, E.P. Duarte Jr., and C.D. Murta

A service provider must continuously monitor the system, in order to guarantee that its multiple concurrent customers are being assigned the amount of resources that guarantees previously agreed on service levels. The provider must deal with diﬀerent customer requirements and must ensure that each customer receives enough resources for each task sent to the system [7]. The provider must evaluate and decide in real time for each task which resources must be assigned for its execution. This is a continuous process: assigning tasks to resources must be evaluated and tasks should be re-scheduled as needed. The process of selecting the right computational elements that will execute a set of tasks is complex, depending on task priority, the capacity of resources and the system load. Implementing such decision and resource allocation process requires a systematic approach for continuously collecting and organizing a set of performance and system load parameters from all system components (e.g. processors or network connections) in order to match resources and tasks eﬀectively and continuously for each task that arrives at the system. This is a complex management process [8] that involves a very large amount of monitoring data, which must be organized, indexed and queried very frequently. This work presents an eﬃcient approach for monitoring service level that keeps track of multiple simultaneous parameters. The monitoring information can be used to decide and match resources and tasks, given a set of service level requirements. A system for mapping information obtained from monitoring k diﬀerent parameters over a k-dimensional space was implemented and tested. Each dimension represents a measure of a quantitative metric related to a parameter of a system resource, e.g. system load, idle processing capacity, amount of free memory or available network bandwidth. The proposed strategy allows multidimensional range search to be performed with cost O(logN ), no matter how large k is, where N is the number of samples or the size of the monitored log. The k-d (k dimensional) tree [9] is used for online monitoring and the k-d-B (k dimensional B) tree [10] is used for oﬄine monitoring. Related work leads to the conclusion that most grid and utility systems collect and store this kind of information in traditional databases [2], [6] and [11]. Applying multidimensional and range search on those systems usually involves complex and potentially slow procedures, and is not feasible especially for on-line monitoring a set of multiple, concurrent system parameters. The proposed approach was implemented using SNMP (Simple Network Management Protocol) [12] for instrumentation. Experimental results are presented which show the eﬀectiveness of the proposed strategies. Multiple resources are concurrently and permanently monitored and alarms can be adjusted to inform that some resources are reaching thresholds or are not fulﬁlling expected goals. Experimental results are shown comparing the proposed approach to a traditional database, for up to 8 million data samples and up to 10 diﬀerent keys, conﬁrming that the database cannot be used for on-line monitoring of multiple parameters. A case study is described which was executed with real data obtained from a Telecom billing system, as well as for a set of Web and video servers.

Monitoring Multiple Concurrent Service Level Parameters

31

The rest of this work is organized as follows. Section 2 presents an overview of related work. In section 3 the data structures employed are speciﬁed, as well as the types of queries supported. Section 4 describes the proposed monitoring strategy. Experimental results and the tool we implemented are described in section 5. Section 6 concludes the paper.

2

Service Level Monitoring

Performance and reliability monitoring is one of the basic functionalities of network management systems [13]. Monitoring systems usually involve a cycle consisting of following steps [14]: (1) resource monitoring data is collected; (2) the obtained data is processed in order to discover resource utilization trends and (3) the system is reconﬁgured, given the conclusions of the previous step. Service management can be seen as a natural evolution of network management, which actually extends traditional performance and availability monitoring strategies to include the concept of service as one of the key managed objects. A service can be provided by several types of entities, and the service deﬁnition varies widely, examples include the execution of a task that requires intensive processing power or the execution of a query on a database at a service provider. It is common to have service levels speciﬁed by Service Level Agreements (SLA). The purpose of service level monitoring systems is to ensure that clients obtain the required service levels while providers optimize resource utilization. A SLA is composed by [15]: a description of the service to be provided, the speciﬁcation of performance and reliability levels required; a detailed contingency procedure, associated monitoring and notiﬁcation procedures; duties and penalties to be applied if the provided service does not meet the required performance levels. Exceptions are also speciﬁed, i.e. situations in which the SLA is not applied. A SLA also includes information such as procedures for performance data collection and metric conversion [2]. Typical metrics include response time and bandwidth available. Monitoring a speciﬁc service given the corresponding SLA involves, besides the SLA speciﬁcation, obtaining information about the service execution, evaluating the service performance and availability, and deciding on future task allocation policies. If the load on servers is heavy, the previously agreed service levels may not be met. In this case the SLA monitoring system must decide, in real time, which agreement should be violated [11]. The amount of information that a service provider has to deal with is huge. This includes information about system resources and services available, their load, client requests, and their requirements as speciﬁed in the SLA. All this information has to be processed in real time, allowing tasks to be scheduled in a way to both meet requirements and use resources eﬀectively. Related work include a number of frameworks and systems proposed recently to monitor and control service level agreements. A SLA monitoring architecture is presented in [16]. This system includes modules for SLA speciﬁcation, server discovery, metric deﬁnition, service monitoring and the detection of agreement

32

A. Kiefer, E.P. Duarte Jr., and C.D. Murta

violation. Frameworks with similar functionalities are described in [2], [6] and [11]. In [11] the authors propose contracts speciﬁed with XML; they aim at helping in the deﬁnition of precise contracts, avoiding the ambiguity that results when natural language is employed. The system provides strict load control in order to avoid the system to get overloaded and, at the same time, maximize the load to optimize system usage. In all these proposals the authors recognize that it is very important eﬃciently collect, store and retrieve service management information. In virtually all related work the authors do not fully specify how these data are managed, i.e. they do not deﬁne a data structure or format for keeping or searching the monitoring data. In [2], [6] and [11], the authors mention the use of traditional database systems. It is important to highlight that the sheer amount of data that needs be stored by a service management system, the types of search that must be executed in order to match resources satisfying speciﬁc criteria in real time can easily represent more than typical database systems are able to handle. In [17], the authors argue that it is not practical to employ a traditional database systems for keeping/searching service monitoring data, where the authors evaluate the high cost of employing SQL queries in databases in this kind of environment. In the next section we present multidimensional data structures that we propose precisely to solve this problem.

3

Multidimensional Search

Several types of applications in diverse ﬁelds require the storage and manipulation of large sets of multidimensional data. Example applications span areas such as Image Processing and Geographic Information Systems. Data can be represented as points, lines, rectangles, areas, volumes, among others on the Euclidean space. The attributes of multidimensional data can be represented as an array. The simplest data unit that can be represented is a point in the k-dimensional space, where k represents it set of attributes. Figure 1 shows an example point in a two dimensional plane. Array (3,5) has value 3 for the x axis and 5 for the y axis.

Fig. 1. A point represented in the bidimensional space

Monitoring Multiple Concurrent Service Level Parameters

33

In the past twenty years numerous data structures have been proposed for keeping and searching multidimensional data eﬃciently. Most of them are variations of multidimensional search trees or k-d trees (k-dimensional trees) [9,18]. The k-d tree is a generalization of a binary tree. The k-d tree receives a set of multidimensional points as input data. These points are embedded in a multidimensional space. As a point is inserted in a k-d tree, it causes a recursive space partition based on the division of hyperplans with k-dimensions. For example, if k=3, then the division hyperplanes will alternate among x, y and z axes. The hyperplanes can be obtained using the mean value of all points in each dimension, which is employed as a discriminant. In this way, each other point with the corresponding key value less than or equal to the discriminant is placed on the left subtree and points with greater values are placed on the right subtree. For each dimension a new discriminant is obtained and the data insertion algorithm is executed recursively until all points have been inserted. Figure 2 shows two common representations of a two-dimensional k-d tree: on a 2-dimensional space and as an abstract tree. In both representations point p1 was chosen as discriminant and it divides the space in two hyperplanes. p1 is the root of the tree. The next division plan crosses p2 (left subtree) and p3 (right subtree). This procedure completes when all points have been inserted in the tree.

p9 p5 p2

p15 p10

p1

p18 p14

p13

p2

p3

p4 p1 p8

p7

p11

p4

p17 p6

p19 p16

p8

p20 p12

p3 p5

p9

p6 p10

p11 p15

p7

p12 p16

p13 p17

p19

p14 p18

p20

Fig. 2. k-d tree: two-dimensional plane (left) and abstract tree (right) representations

The k-d tree is a data structure that has to be entirely kept in main memory. In case the tree is too large and must be stored in secondary memory, algorithms will require paging, i.e. obtaining tree chunks from disk. As the tree grows increasingly unbalanced, performance will deteriorate, and logarithmic algorithms may become linear. An alternative to solve this problem is to employ another balanced multidimensional tree meant for secondary memory, that is the k-d-B tree proposed by Robison [10]. The k-d-B tree can be seen as variation of the k-d tree that combines the logarithmic algorithms of k-d tree with the eﬃcient secondary memory management algorithms of B trees. Thus it is eﬃcient to process a tree that is not completely loaded to main memory. The k-d-B tree is a multi-way tree that keeps all leaves at the same level. Like the k-d tree, the k-d-B tree also partitions the multidimensional space on which data are embedded in sub-spaces. Each internal node

34

A. Kiefer, E.P. Duarte Jr., and C.D. Murta

corresponds to a rectangular region and its children deﬁne a disjoint partition of that region. The points are stored in the leaves of the tree. Figure 3 shows an example k-d-B tree. Each internal node represents a rectangular region in the planar representation, and each leaf represents a partition obtained by recursive decomposition of the x and y axes using parallel lines (x1, x2, x3 and x4; y1, y2 and y3). x2 p2 p5

y1

p7

p15

p9

p4

x4 p18

p13

p10

p14

p3

y2

p17

p11

p1

p6

p20

p19

p8

p16 p12

y3

x3

x1 x2

p7

p15

p9 p2 p5

p13

p10

y1 p4

p8

p11

p1

x1 p2 p5 p9

x4

p17 p6

p6 p11

P20

p16 p12

p14

p3

y2

y2

y3

x3

x1

p1 p4 p8 p10 p15

p19

p18

p17 p19 p20 p12 p16

p3 p7 p13 p14 p18

Fig. 3. A k-d-B tree planar representation

3.1

Multidimensional Search Types

Searches are typically initiated in response to a query. A search algorithm returns the records that match the query condition. A multidimensional search, sometimes also referred as associative search or search using secondary keys operates on records with several keys speciﬁed as tuples of attributes. A record can be represented in the Euclidean space as a point, its attributes correspond to the point’s coordinates in the space. A tuple represents coordinates in a multidimensional space, where each dimension represents one attribute. A query retrieves all records that satisfy some properties. Various types of queries exist [19] and are classiﬁed according to speciﬁcation of desired records, also called a region. A query always retrieves all records that fall within the speciﬁed region. A region is deﬁned as a set of maximum and minimum coordinate values in the geometric space. A description of the main types of queries follows. The exact match is the most simple query, and returns a speciﬁc record deﬁned by its k keys. A partial match yields a set of registers whose k keys match at least one key speciﬁed in the query. The range match looks for all records that have their k keys within the speciﬁed ranges of keys of key values. Finally, a proximity query looks for the records that are closest to a given set of key values.

Monitoring Multiple Concurrent Service Level Parameters

4

35

The Monitoring Strategy

The proposed monitoring strategy is described in this section. Multiple concurrent management objects provide automated data collection of selected system resources, for example, CPU, disk, memory, network interfaces. The monitoring strategy is composed of three phases: data sampling, data insertion into the multidimensional space, and requirements evaluation (assessment of requirements). The monitored parameters are SNMP objects. These objects collect performance data and quantitative information of the computing device monitored. Each object is associated with requirements described by arithmetic and logic functions. Multiple requirements reﬂect restrictions on the values of the objects. To carry out the ﬁrst phase, sampling, we deﬁne the monitoring objects, the SNMP agents from which objects are derived, as well the sampling interval. In the second phase, the sampled values are mapped to a multidimensional space. In this space, each dimension corresponds to an object. Each point of this space is a managed computing resource or device. Figure 4 shows an example of a bidimensional space. Four computing units are monitored and mapped in a bidimensional space that represents two computing resources: available memory and processor utilization. In this picture, C1, C2, C3, and C4 represent the resources monitored. The available memory is plotted on the x axis, while the processor utilization is represented on the y axis. The points M1, M2, M3, and M4 are values for available memory (%) for each of the four resources C1, C2, C3, and C4. In the same way, P1, P2, P3, and P4 are the processor utilization (%) of each computing resource.

Fig. 4. Mapping of objects and resources in a bidimensional space

36

A. Kiefer, E.P. Duarte Jr., and C.D. Murta

The mapping of the objects in the multidimensional plane is done with multidimensional search trees in the second phase. The k-d tree is the data structure chosen for online monitoring, while the k-d-B tree is provided for oﬄine monitoring. The search in both trees can be based on speciﬁed computing resource metrics. In the third phase, the monitoring strategy retrieves data from the updated multidimensional search tree. The queries are built on the requirements. The multidimensional search is applied to all monitored resources. The multidimensional search returns the devices that match the speciﬁed performance parameters and based on thresholds a warming or alarm can be issued if requirements are not met. The tool can be also record the time interval in which resources presented a given performance. A timestamp is appended to the collected data items, so it is possible to evaluate the history of the resource behavior.

5

Monitoring Tool Description and Experimental Results

In this section we describe a tool that implements the proposed strategy and the experimental setup and results. The tool is composed by a monitoring module which is based on SNMP, and a module to manage the multidimensional trees. 5.1

Description of the Monitoring Tool

The tool was written in C++ and runs on the Linux operating system. The data collection module is based on Net-SNMP[20]. The tool obtains data from SNMP agents. Some SNMP objects are directly used, without any extra processing. Other SNMP objects, e.g. counters, require two sequential samples to be obtained in order to give meaningful information. Consider for instance how the processor load and memory usage are monitored. The processor load is computed from objects ssCpuRawNice, ssCpuRawUser, ssCpuRawSystem, and ssCpuRawIdle, availabled in UCDSNMP MIB [20]. The values represent, respectively, the time the system was in “low priority”, “ user”, “system”, and “idle” modes, all of which are kernel measures. Memory usage is computed from objects hrStorageSize and hrStorageUsed implemented in the Host Resources MIB [21]. These objects represent, respectively, the amount of available memory and used memory. All samples collected are stored in secondary memory before being handled to the multidimensional information management module, described below. The multidimensional information management module is responsible for the insertion the collected samples in multidimensional search trees. The k-d-B tree was implemented using the toolkit TPIE (Transparent Parallel I/O Environment) [22], that allows the implementation of external memory algorithms, and minimizes the input and output communication (I/O) performed when solving

Monitoring Multiple Concurrent Service Level Parameters

37

problems on very large data sets. The k-d tree was implemented using the template libkdtree [23]. Both the k-d-B tree and the k-d tree implementations allow data to have, at least in theory, an unlimited number of dimensions.

5.2

Results

We initially present results comparing the proposed approach with a traditional database, in this case MySQL. We measured the time required for inserting and searching up to 8 million monitoring samples. The comparison was performed for oﬀ-line monitoring, and thus only with k-d-B trees, which also involve disk accesses. It is important to remark that k-d trees used for on-line monitoring is orders of magnitude faster than both databases and k-d-B trees. In our experiment the database either presented a very high search time or a very high index building time. In order to get roughly the same search time of multidimensional trees (ranging from 2 to 6 milliseconds) the database had to be conﬁgured to build indices for all 10 keys. Figure 5 shows the insertion time measured in this case. While in our approach the time required is at most a few seconds, the database required more than 2 hours for fully indexing 8 million records.

insertion - 10 keys 2.5e+07 mysql kdbtree

time (ms)

2e+07

1.5e+07

1e+07

5e+06

0 0

1e+06 2e+06 3e+06 4e+06 5e+06 6e+06 7e+06 8e+06 # records

Fig. 5. Comparison of the insertion time: k-d-B tree and MySQL

38

A. Kiefer, E.P. Duarte Jr., and C.D. Murta

We now present experimental results obtained from monitoring three Web servers and three video servers. In the experiments, six parameters were sampled: the time instant the SNMP query was issued; the utilization of the input network interface; the utilization of the output network interface; memory usage; processor utilization; and number of active TCP connections. Servers were connected to a non-dedicated 100Mbps Ethernet network. The monitoring samples were inserted into the k-d tree. Three experiments were then executed with diﬀerent purposes. The ﬁrst experiment was meant to show that the tool performed as expected and that the multidimensional search returned correct values. The second experiment was performed to show that the variation in the number of key attributes did not aﬀect the tool performance. The third and ﬁnal experiment show the relationship between the response time of the multidimensional search and the number of records stored in the k-d tree. The response time is an important parameter to allow the tool to be used for continuous on-line monitoring. In the ﬁrst experiment, a video server was monitored for an interval of 60 seconds. Samples containing the six monitoring parameters were collected every second and inserted in the k-d tree. Figure 6 shows results. Several multidimensional range queries were executed including (ranges are speciﬁed in brackets): processor utilization: [5-7] %, output network interface utilization: [1525] %, memory usage: [85-95] %. For the other parameters, the range of query was the range of the ﬁeld of records. The multidimensional search is shown in Figure 6. The horizontal lines show the speciﬁed limits for the three parameters. The query retrieved 38 records that meet the criteria described above, which means that during approximately half of the time of observation, the selected parameters presented their values within the range speciﬁed by the search. This result conﬁrms that the tool produces accurate results as expected, i.e. it allows monitoring and range search for system parameters according to speciﬁed values. The goal of the second experiment is to evaluate the relationship between the number of query attributes, that is, the search keys, and the response time of the search in the k-d tree. The video server and the Web server were monitored. The monitoring phase has produced 600 records with 4 keys, 600 records with 5 keys, and 600 records with 6 keys. A tree was built for each set of 600 records and the insertion time was measured. The experiment was repeated hundreds of times. Table 1 presents the average insertion time and the average search time measured in this experiment with a varying number of keys. It is easy to see that the insertion time is constant, i.e. does not vary with the number of search keys. This result is expected as the multidimensional search tree is a generalization of the binary search tree. At each tree level, one key is used as the search parameter, and guides the traversal. Another consideration is that the time to search the tree does not depend on the type of information stored, i.e. it is independent of the content of records and, consequently, independent of the kind of monitored resources. The result of this experiment conﬁrms that we are able to issue queries using a large number of keys. As a consequence, we can

Monitoring Multiple Concurrent Service Level Parameters

39

100

Memory Network (out) CPU

Resources Utilization (%)

80

60

40

20

0 0

10

20

30

40

50

60

Time (s)

Fig. 6. Utilization of the video server resources Table 1. Average time of insertion and search in a k-d tree using 4 to 6 search keys # keys # records Average time # records Average time inserted insertion (ms) retrieved search (ms) searched 4 600 20 600 20 5 600 20 600 20 6 600 20 600 20

record and retrieve several system resources simultaneously with no penalties for keeping the chosen data structure. The third experiment is designed to evaluate the search time in the multidimensional search tree as a function of the number of records entered and obtained from the experiment. In this experiment, the servers were monitored by a period of 30 minutes (1,800 seconds), and samples were collected every second. These samples were inserted into the multidimensional search tree every 60 seconds. This procedure was repeated until the end of the experiment. For each insertion the time (in ms) of the operation was measured, as well as the time (in ms) to execute the range search. The range was deﬁned in a way to allow the retrieval of all data stored, which is the most comprehensive search possible. The response times for the insertion of samples collected from the start of monitoring (generating a history of monitoring), and the response times to search the multidimensional trees with a varying number of records inserted are shown

40

A. Kiefer, E.P. Duarte Jr., and C.D. Murta 100 Insert Range Search

Time (ms)

80

60

40

20

0 0

200

400

600

800

1000

1200

1400

1600

1800

Records

Fig. 7. Insertion and search response times as a function of the number of records in tree

in Figure 7. This Figure shows that both the insertion time and the response time grow with the number of records. For example, when we have 230 records, the response time for the insertion operation is around 10 ms. In order to give an idea of the time limits presented for on-line monitoring we can add the insertion time plus the multidimensional range search time, we obtain a response time of 20 ms. In this case, the proposed strategy takes 20 ms on average to insert 230 records in a multidimensional search tree and to get the result of a query in this data structure. Consider a new request that arrives and has to assigned to a computational unit, selected in terms of performance parameters. For each new request, a search is executed in about 10 ms. For 100 requests, the response time of the search would is about 1 second, which results in a rate of approximately 100 requests answered by second. The response time of the query by interval may be reduced if the number of records stored in the tree is also reduced, this can be done by entering only the most recent records in the tree, which will reﬂect the recent state of the system. We also evaluate the proposed tool using Call Detail Records (CDR) of a real telecom billing system. The call processing system evaluated produces an average load of 300 CDRs/s. Each CDR generates 1KB of data. The system under evaluation processes an average of 60,000 calls/s, and the CDR are collected at every three minutes. We have collected 100,000 CDR, that accounts for approximately ﬁve minutes of system activity. We inserted these records in the multidimensional data structure under test. The results shows that the insertion time as well as range search are logarithmic, as expected. This experiment conﬁrms that the data structure can be eﬀectively used to handle real-world systems data.

Monitoring Multiple Concurrent Service Level Parameters

6

41

Conclusion

In this paper we presented a strategy to eﬃciently monitor multidimensional data which consists of several parameters. Data instrumentation is based on SNMP. Queries can include multiple concurrent requirements on several system devices. The multidimensional search can be used for instance to ﬁnd resources that meet speciﬁc requirements within a set of monitored distributed systems. The proposed strategy is based on storing data in multidimensional trees which allow logarithmic range search. A implementation was described that allows both continuous online monitoring with the k-d tree and oﬄine monitoring massive amounts of logged data with a k-d-B tree. Experimental results are shown comparing the proposed approach to a traditional database. The comparison involved up to 8 million data samples and up to 10 diﬀerent keys, conﬁrming that the database either requires a time consuming full index creation or presents very high search delays. A set of Web and video servers was monitored. Server resource information was collected and inserted in the data structures, allowing the identiﬁcation of periods in which the servers were heavily/lightly used during the period of observation. Overall the results conﬁrm that the proposed strategy can be eﬀectively applied to monitor real systems. Besides including the development of a Web interface for the tool, future work is focused on integrating the proposed tool to a cloud computing platform.

Acknowledgements The comparison of our approach to a traditional database (MySQL) was possible thanks to the implementation and careful evaluation conducted by Saulo Quinteiro dos Santos, Renato Yamazaki, and Luiz F. A. de Pr´ a, at UFPR. This work was partially supported by grant 311221/2006-8 from the Brazilian Research Agency (CNPq).

References 1. Lawton, G.: Moving the OS to the Web. Computer 41(3), 16–19 (2008) 2. Buco, M.J., Chang, R.N., Luan, L.Z., Ward, C., Wolf, J.L., Yu, P.S.: Utility Computing SLA Management based upon Business Objectives. IBM Systems Journal 43(1), 159–178 (2004) 3. Livny, M., Raman, R.: Enterprise Resource Management: Applications in Research and Industry. In: Foster, I., Kesselman, C. (eds.) The Grid, 2nd edn. Morgan Kaufmann, San Francisco (2003) 4. Kumar, V., Schwan, K., Iyer, S., Chen, Y., Sahai, A.: The state-space approach to SLA-based management. In: 11th IEEE/IFIP NOMS (2008) 5. Sall, M., Bartolini, C.: Management by Contract. In: 9th IEEE/IFIP NOMS (2004) 6. Bouillet, E., Mitra, D., Ramakrishnan, K.G.: The Structure and Management of Service Level Agreements in Networks. IEEE Journal on Selected Areas in Communications 20(4), 691–699 (2002)

42

A. Kiefer, E.P. Duarte Jr., and C.D. Murta

7. Abraho, B., Almeida, V., Almeida, J.: Self-Adaptive SLA-driven Capacity Management for Internet Services. In: 17th IEEE/IFIP DSOM (2006) 8. Taylor, R., Tofts, C.: Death by a Thousand SLAs: A Short Story of Commercial Suicide Pacts (2006) 9. Bentley, J.L.: Multidimensional Binary Search Trees Used For Associative Searching. Communications of the ACM 18(9), 509–517 (1975) 10. Robinson, J.T.: The K-D-B-Tree: A Search Structure for Large Multidimensional Dynamic Indexes. In: Lien, Y.E. (ed.) Proceedings of the 1981 ACM SIGMOD International Conference on Management of Data, Ann Arbor, Michigan, April 29 - May 1, pp. 10–18. ACM Press, New York (1981) 11. Leﬀ, A., Rayﬁeld, J.T., Dias, D.M.: Service-Level Agreements and Commercial Grids. In: IEEE Internet Computing, July-August 2003, pp. 44–50 (2003) 12. Harrington, D., Presuhn, R., Wijnen, B.: An Architecture for Describing Simple Network Management Protocol (SNMP) Management Frameworks. Request for Comments 3411 (2002) 13. Stallings, W.: SNMP, SNMPv2, SNMPv3 and RMON1 and 2, 3rd edn. AddisonWesley Longman Publishing Co., Inc., Amsterdam (1998) 14. Leinwand, A., Conroy, K.F.: Network Management a Pratical Perspective, 2nd edn. Addison-Wesley Longman Publishing Co., Inc., Amsterdam (1996) 15. Verma, D.: Supporting Service Level Agreements on IP Networks. Macmillan Technical Publishing, Basingstoke (1999) 16. Molina-Jimenez, C., Shrivastava, S., Crowcroft, J., Gevros, P.: On the Monitoring of Contractual Service Level Agreements (2004) Technical Report series CS-TR835, School of Computing Science, University of Newcastle upon Tyne (April 2004) 17. Dinda, P.A., Lu, D.: Nondeterministic Queries in a Relational Grid Information Service. In: SC 2003: Proceedings of the 2003 ACM/IEEE Conference on Supercomputing, Washington, DC, USA, pp. 12–26. IEEE Computer Society Press, Los Alamitos (2003) 18. Bentley, J.L.: Multidimensional Binary Search in Database Applications. IEEE Transactions on Software Engineering 4(5), 333–340 (1979) 19. Gaede, V., Gunther, O.: Survey on Multidimensional Access Methods (Revised Version). ACM Computing Surveys 30(2), 170–231 (1998) 20. The NET-SNMP Project Home Page (2003) , http://net-snmp.sourceforge.net, (accessed in, July 2009) 21. Waldbusser, S., Grillo, P.: Host Resources MIB. Request for Comments 2790 (2000) 22. A Transparent Parallel I/O Environment (2003), http://www.cs.duke.edu/TPIE/, (accessed in, July 2009) 23. The libkdtree++ Project (2004), http://freshmeat.net/projects/libkdtree/, (accessed in, July 2009)

A Performance-Based Methodology to Improve Grid Exploitation A. Clematis1, A. Corana2, D. D'Agostino1, A. Galizia1, and A. Quarati1 1 IMATI-CNR, Via De Marini 6, 16149 Genova, Italy {clematis,dagostino,galizia,quarati}@ge.imati.cnr.it 2 IEIIT-CNR, Via De Marini 6, 16149 Genova, Italy [email protected]

Abstract. Due to their complexity, the exploitation of Grid environments is not a trivial activity for many users, and a key factor is to enable a simplified and transparent orchestration of resources and jobs. Particularly critical is the deployment of matching procedures capable to effectively meet user’s requirements with resources offer. We introduce GREEN a management tool primarily devoted to the matchmaking process, based on a performance characterization of both resources and job requirements. Leveraging on a two-level benchmarking methodology, GREEN allows users to express performance preference through an appropriate extension to Grid submission and description languages such as JSDL and Glue. Operating at intermediate level between applications and Grid middleware, GREEN reduces the gap between users’ needs and available resources thus enabling a seamless exploitation of the Grid. Keywords: Grid management, Benchmark-driven matchmaking, Grid language extensions.

1 Introduction Grid environments are service-oriented infrastructures that facilitate the sharing of instruments, knowledge, data and computational resources managed by different organizations in widespread locations and supply their exploitation through the submission and the execution of users’ jobs. Since their first appearance, Grids showed great potentialities for the scientific community as they allow the definition of virtual spaces providing huge computational power and collaboration tools to scientists [1]. Examples of Grid adoption in the scientific realm are found in projects such as CaBIG, Worldwide LHC Computing Grid, AstroGrid investigating respectively bioinformatics, high-energy physics and astronomy issues [2-4]. The Grid is also exploited to guide business experiments, for example, the Business Experiments in Grid project (BEinGrid) is aimed to highlight scenarios, solutions and results in 25 case studies [5]. To support experiments and investigations, distributed resources and jobs have to be orchestrated in such a way that user’s objectives are addressed without requiring a deep and difficult interaction with the resources. Actually, the consumption of the shared resources in a Grid could be not trivial as they are heterogeneous and generally C. Bartolini and L.P. Gaspary (Eds.): DSOM 2009, LNCS 5841, pp. 43–55, 2009. © IFIP International Federation for Information Processing 2009

44

A. Clematis et al.

belong to different Physical Organizations (POs). POs are subject to a variety of configuration settings and are usually federated in Virtual Organizations (VOs). VOs group people with similar interests and aims, thus leading to the identification of sets of common (i.e. most used) applications, each owning specific requirements and execution modes [6]. Due to the organizational and technological complexity of these environments, practices and tools to manage resources and to model and maintain their consistent description are required. In particular, information about resources properties, their current state and user’s specific requirements is essential to guarantee that a job submitted by a user will be forwarded to the most appropriate resource. Indeed, this supply-demand coupling process is a critical one, since it reflects on the effective execution of each distinct user’ application, and significantly impacts on the overall performance of a VO as a whole [7]. The responsibility for performing this crucial activity is commonly left to the matchmaking component, whose main task is to grant the discovery of available resources and services on the base of the specific properties defined by users and expressed through pertinent requests [8]. To this end, the matchmaker may greatly benefit from a performance characterization of resources based on the employment of benchmarks [9]. Benchmarking represents a powerful mean to investigate, characterize and compare the performance of different computer systems in order to select the most suitable resource to execute a class of applications. Considering traditional microprocessors as well as High Performance Computing systems, it is possible to outline two categories: Micro-benchmark and Application-specific benchmarks. The former is apt to profile resources considering isolated low-level capabilities such as CPU, memory, and interconnection speed [10]. The latter is apt to stress simultaneously several aspects of the system, and corresponds to the computationally demanding part of real applications. Moving towards Grids, the characterization of computational resources through benchmarks is largely acknowledged together with its intrinsic criticality [11,12], mainly due to the multi-layered, dynamical, heterogeneous structure of the Grid, and often hindered by the specific procedures adopted by each VO in classifying and making resources accessible. In this paper, we present GREEN, Grid Environment ENabler, a management tool designed to assist Grid administrators and users to set-up, administrate and exploit Grid infrastructures, with prior activity the matchmaking process. To fulfil this goal, GREEN relies on a two-level benchmark methodology, i.e. Micro and Application-specific, through which every resource of a PO is tagged with the performance results obtained under different workloads. Operating at intermediate level between applications and Grid middleware, GREEN focuses on the discovery of resources satisfying user requirements ordered by performance ranking, while the selection of any particular amongst them, is left to a scheduler, responsible to apply the proper policies. To sustain the matching operation, GREEN offers administrators and users, functionalities to store benchmarks results and to submit jobs, respectively. From the administrator’s point of view, GREEN supports the creation and maintenance of the performance description, allowing to efficiently respond to user’s requests of integrating new relevant application-driven benchmarks. From user’s point of view, GREEN enables the declaration of a ranking preference for the resources during job submission, i.e. the selection of the benchmark to guide the matching process. GREEN receives requests of job submission

A Performance-Based Methodology to Improve Grid Exploitation

45

initiated by users; it uniforms execution requests, expressed through different Job submission languages, thus addressing interoperability issues; and carries-out their subsequent submission to the underlying middleware. The outline of the paper is as follow. Section 2 discusses some valuable contributions in the fields of matchmaking and benchmarking on Grid. In Section 3, we present our two-level benchmarking methodology along with some preliminary results highlighting its appropriateness in Grid scenarios. Section 4 introduces GREEN as a management tool for Grid environments, focusing on a technical overview. Section 5 gives some concluding remarks.

2 Related Works The implementation of an efficient and automatic mechanism for the effective discovery of the resources that best fit the requirements of users’ job is one of the major problems in present Grids. A possible way to improve the efficiency of this step is to drive the search towards resources that show good performance in the execution of jobs with similar or known behavior. This issue initially obtained little attention from the middleware designers and developers, thus several projects and tools proposed solutions to address the topic. From the middleware point of view, the Globus toolkit did not provide, originally, a resource matchmaking/brokering as core service. However, since June 2007, the GridWay metascheduler [13] has been included in the Globus distribution as an optional high-level service. GridWay allows users to specify a fixed and limited set of resource requirements, mainly related to the queue policies of the underlying batch job systems. Benchmarks are not considered at all, and this choice limits the ranking of resources. On the contrary, gLite has a native matchmaking/brokering service that takes into account a richer set of requirements, including benchmark values. This service is based on a semi-centralized approach, and may result in long waiting time in the job execution. The set of benchmarks actually considered by gLite, i.e. the SPEC suite, mainly evaluates CPU performance [14]; thus, the description of system performance may result partial, hence not completely suitable to specific application requirements. A more accurate strategy should take into account some of the proper characteristics of the application at hand, as claimed in Section 3. Due to the peculiar nature of the Grid, performance evaluation in a dynamical, heterogeneous context is more complex and less deterministic than in traditional scenarios. In fact, the Grid has a multi-layered structure, thus benchmarks investigating performance aspects of the different Grid layers should be considered in order to grasp a predictable behaviour of a real application run [11]. Actually, besides the set of interesting parameters to measure the single isolated resource, e.g. CPU speed, memory and interconnection bandwidth, different factors have to be taken into account when considering the execution of a benchmark (suite) on Grid. For example, the Grid Assessment Probes [15] has a means of attempting to provide an insight into the stability, robustness, and performance of the Grid. The probes are designed to serve as simple Grid application exemplars and diagnostic tools. They test and measure performance of basic Grid functions, including file transfers, remote execution, and Grid Information Services response. GridBench [9] is a complex and interesting

46

A. Clematis et al.

tool that provides a graphical interface to define, execute and administrate benchmarks. It takes into account interconnection performance and evaluates resource workload, and can be used to rank Grid resources. The NAS Grid Benchmark (NGB) suite [16] is defined by NASA, and represents typical activity of Computational Fluid Dynamics applications. It provides a set of computationally intensive benchmarks representative of scientific, post-processing and visualization workloads, and tests the Grid capabilities to manage and execute distributed applications. A brokering mechanism based on benchmarking of Grid resources is proposed by Elmroth and Tordsson [17]. However, the scope of the broker is focused on the ARC middleware and the NorduGrid and SweGrid production environments, and it adopts an extension of RSL (earlier Globus submission language) to submit user’s jobs, conversely to our proposal aimed to follow a more interoperable approach.

3 A Two-Level Benchmarking Methodology To describe Grid resources, we propose a two-level methodology aimed to give a useful enriched description of resources, and to facilitate the matchmaking process. Our methodology considers two approaches: I) the use of micro-benchmarks to supply a basic description of resource performance; II) the deployment of application-driven benchmarks to get a closer insight into the behavior of resources under more realistic conditions of a class of applications. Through application-driven benchmarks, it is possible to add an evaluation of the resources based on the system indicators that are more stressed by an application. 3.1 Micro-Benchmarks In order to supply a basic resource characterization, mainly based on low-level performance capacity, we considered the use of traditional micro-benchmarks. To this aim, a reasonable assumption is that the performance of a machine mainly depends on the CPU, the memory and cache, and interconnection performance [10]; therefore, we individuated a concise number of parameters to evaluate in order to provide an easyto-use description of the various nodes. We selected a set of five, largely widespread, benchmarks able to capture relevant metrics to characterize computational resources’ performances. In particular, Flops provides an estimate of peak floating-point performance (MFLOPS) by making maximal use of register variables with minimal interaction with main memory [18]. Stream is the industrial de facto standard benchmark to measure sustained memory bandwidth [19]. CacheBench is designed to evaluate the performance of the memory hierarchy of computer systems, expressed by raw bandwidth in megabytes per second [20]. Mpptest measures the performance of some of the basic MPI message passing routines in a variety of situations [21]. Bonnie performs a series of tests on a file of known size. For each test, it reports the bytes processed per CPU second, and the percentage of CPU usage [22]. The micro-benchmarks used in this phase generally return more than a value. To obtain results easily usable in the matchmaking process, we considered for each benchmark synthetic parameters or the most significant value. These results are managed by GREEN to populate the benchmark description of resources.

A Performance-Based Methodology to Improve Grid Exploitation

47

3.2 Application-Specific Benchmarks Micro benchmarks are a good solution in the case of applications stressing mainly one architecture aspects, e.g. CPU intensive, or not frequently executed. Indeed, usually the participants to a VO have similar aims, from which a set of the most used applications emerges. In these cases, a more suitable approach is to evaluate system performance through application-specific benchmarks that approximate at best the real application workload. This benchmarking level offers two procedural approaches a) the use of a “light” version of the application at hand, with a reasonable computational cost but still representative of the real behaviour; b) the use of well known application specific benchmarks largely employed in the scientific community. As case studies, we considered some applications of our interest, i.e. image processing, isosurface extraction, and linear algebra. For the first two classes of applications, we adopted approach a) using a sequential code aimed to emphasize precise aspects of the considered metrics. With respect to image processing, we selected a compute intensive elaboration applied to a reference image of about 1 MB; in this way, CPU metrics are mainly stressed. Hereafter we refer to this code as Image Processing Benchmark (IPB). The isosurface extraction application provides a more exhaustive performance evaluation of the system, as it also heavily involves I/O operations. In this case, we considered the processing of a small 3D data set of 16 MB, producing a result of 67 MB. Following approach b) for the class of applications based on linear algebra, we selected the well-known High Perfomance Linpack (HPL) benchmark [23]. For applicationdriven benchmarks, the metric considered to characterize resources is wall clock time. Similarly, to the micro-benchmarks case, the results are stored in the internal data structure of GREEN. 3.3 Methodology Evaluation To evaluate the effectiveness of our methodology, we experimented upon two specific resources: 1) a Beowulf Cluster made up of sixteen nodes interconnected by a Gigabit switched Ethernet. Each node is equipped with a 2.66 GHz Pentium processor, 1 GB of RAM and two EIDE disks interface in RAID 0 2) the SiCortex SC1458 system with 243 SiCortex node chips, each equipped with six cores; linked by a proprietary interconnection network supporting large message bandwidth of 4 GBytes/sec. This system pursues the Green Computing guidelines, through extremely low energy consumption. By a quick comparison, clearly emerges that the two resources vary greatly both in terms of the number of CPUs and in terms of individual CPU performance. In fact, SC1458 has a greater number of CPUs than the Beowulf Cluster, but the latter has faster CPUs and better memory bandwidth. Notwithstanding from these technical differences, one may infer consequent performance results, this expectation is contradicted by our experiments. Starting from micro-benchmark results, the SC1458 achieves better performance in almost each case and parameters evaluated, when considering aggregate computing power. However, its single cores have relatively low performance compared with the single CPU of the Beowulf Cluster, and the actual power of the resource derives from the high number of provided cores and the native fast connection among processes. To outline CPU performance, we depicted in Figures 1 and 2 the results obtained with FLOPS and STREAM.

48

A. Clematis et al.

25686 SiCortex.SC1458

Resources

183

Aggregate Power Single CPU

6819 Beowulf Cluster 426

MFLOPS

Fig. 1. Comparison between resources according to FLOPS

Both benchmarks have been run on a CPU/core independently, and then the aggregated results are gathered to represent the performance of the whole parallel resources [9]. For each resource, we present the evaluation of the single CPU/core and the parallel resources.

62916 SiCortexSC1458

Resources

449

Aggregate Power Single CPU

19005 Beowulf Cluster 1188

MB/Sec

Fig. 2. Comparison between resources according to STREAM

Also with respect to interconnection evaluation, the SC1458 achieved definitely better performance, as reported in Figure 3. We tested point-to-point communication performance, through the MPPTest benchmark; results are expressed in MB/Sec. As mentioned above, the Beowulf Cluster employs a Gigabit Ethernet, while SC1458 has a proprietary interconnection that performed significantly better.

A Performance-Based Methodology to Improve Grid Exploitation

49

140000

120000

Beowulf Cluster SiCortex.SC1458

Time (µs)

100000

80000

60000

40000

20000

0 0

1

2

3

4

5

6

7

8

Size (Mbytes)

Fig. 3. Comparision wrt MPPTest

Considering the second level of benchmark, the situation is quite different. In fact, depending on the application domain, better results were obtained alternatively by both resources. We conducted our tests by using IPB and HPL benchmark, and considering the execution times (Wall Clock Time) as metric to evaluate performance. The results are normalized according to a base value; to this end, we adopted the values returned from the Beowulf Cluster. Table 1 reports the values obtained for IPB and HPL benchmark. As already said, in the latter case, we considered all available processes for the Beowulf Cluster, i.e. 16 nodes, while for the SC1458 resource we examined separately the use of different number of processors (16, 64, 128). Table 1. Comparison of executions performance, normalized wrt Beowulf Cluster

IPB HPL benchmark

Beowulf Cluster 1 1

SC1458

SC1458 16 p

SC1458 64 p

SC1458 128 p

0.44

0.13

0.08

6.1

The first row of Table 1 shows that Beowulf Cluster performed significantly better considering the image processing application, but the situation is exactly the opposite for HPL benchmark as expressed in row 2, which highlights that SC1458 outperforms Beowulf up to a factor 10, when increasing the number of processes. This behaviour depends on the different requirements of the two applications. In the analyzed cases, IPB solely benefits from fast single CPU, while HPL tests the entire system and benefits from high number of processes linked with fast connections. Starting from these remarks, it is quite evident that the Beowulf Cluster is faster in the execution of IPB, while it poorly performs with respect to HPL. On the contrary, with respect to HPL, SC1458 outdoes the Beowulf Cluster, but it does not achieve good results on the proposed image processing operations.

50

A. Clematis et al.

Following our methodology, it clearly emerges the differences in the performance of both resources in each level of benchmark. SC1458 definitely performs better than the Beowulf Cluster with respect to the micro-benchmark. However, considering the second level of benchmark, the Beowulf Cluster appears as the suitable choice for the execution of specific applications. This performance divergence also occurred in other similar comparisons we conducted for all the other benchmarks previously described, and thus testifies the appropriateness of our approach.

4 GREEN a Benchmark-Based Tool to Manage Grid Resources To reduce the gap between users and resources, we designed GREEN, a Grid management tools mainly aimed to perform matchmaking based on a performance characterization of resources and jobs. GREEN bases on a distributed approach and leverages on a overlay network infrastructure to connect the various POs constituting a Grid [24]. GREEN introduces some features able to satisfactory fulfil the diverse needs of Grid stakeholder: • • • •

Insertion of benchmark information by system administrators; Supporting users to the submission of Job to the Grid; Translation of job submission expressed into a JSDL document into the specific submission language accepted by the middleware; Execution of the (distributed) matchmaking process;

These functionalities rely on a proper description of resources required both on the job/user and on the owner side, necessary to accomplish the coupling task. In fact, according to our methodology, benchmarking outcomes are used to annotate (tagging) Grid resources. These tags are then compared with the benchmark-related requirements, contained in the job documents submitted by users. Analysing the main success proposals, carried out by different projects and research groups in the field of resources and job description, and aimed to deal with different middlewares transparently to Grid users, we defined two extensions capable of capturing the benchmark characterization of both resources and jobs. 4.1 Extending Languages for Job and Resource Characterization As to resources characterization, we adopted the Grid resources vision offered by the Glue 2.0 specification language [25], which foresees that benchmark-value copies are represented as Glue entities according to the XML reference realizations of Glue 2.0 [26]. By employing the openness of BenchmarkType_t, the set of recognized benchmarks is extensible without any change to the document schema. This solution allows the seamless insertion of new benchmarks data as soon as they should appear relevant to the users of a VO. The specificity of our two-level methodology is modelled with the extension mechanism defined in Glue. We enriched the Benchmark_t type adding the BenchLevel element to specify the benchmark level (i.e. two string values micro and application). An excerpt from a document related to the execution of

A Performance-Based Methodology to Improve Grid Exploitation

51

micro-benchmark Flops against the Beowulf Cluster, whose head node has IP 150.145.8.160, resulting in 480 MFlops is: 150.145.8.160 MFlops 480 micro Listing 1. Example of the extension to the Benchmark element

The counterpart of benchmarking resources is the ability for users submitting a job to express their preferences about the performance of target machines. A job submission request, in addition to stating the application-related attributes (e.g. name and location of source code, input and output files), should express syntactic requirements (e.g. number of processors, main memory size) and ranking preferences (if any) to guide and constraint the matching process on resources. To this end, some mechanism is required to allow users to explicitly assess these requirements inside the job submission document. The three main Job Submission Languages (JSL) currently used by Grid community are the Globus Job Description Document (JDD) [27], the EU-DataGrid Job Description Language (JDL) [28], and the Job Submission Description Language (JSDL) [29] proposed by one of the Working Group of Grid Forum. Evaluating their major properties and how they differentiate each others, e.g. in the support to express requirements on resource, we decided to extend JSDL, whose mission is to provide a standard language to be used on top of existing middlewares. Augmenting JSDL schema to take into account ranking specification, we introduced an element Rank (of complex type Rank_Type) devoted to this task. To maintain a desirable, although not mandatory, uniform lexicon between the JSDL constructs on job side and the Glue description on resource side, we borrowed from the Glue extension the definition of BenchmarkType_t, which is embedded as sub-element of Rank. 4.2 Components Description GREEN is designed as a Grid service based on a distributed and cooperative approach for Grid resource discovery and ranking. For every PO in a Grid, a GREEN instance is responsible for the management of updated data about the state of its resources, and for its exchange with other GREEN instances to satisfy user’s requests. Figure 4 depicts the main components of GREEN, along with some interactions with other middleware services, notably the Information Service (IS) and Execution Environment (EE), occurring after the submission of a job. In the following, we summarise the role and the behaviour of those components: •

The Job Submission (JS) component is the main gateway to GREEN functionalities; it receives requests of benchmark submission by PO administrator or jobs submission initiated by users. Depending on the activation mode (according to

52

•

•

•

A. Clematis et al.

the different published signatures), it behaves just like a messages dispatcher or a translator of JSL documents carrying-out their subsequent submission to the EE, thus addressing interoperability issues. The main task of the Benchmark Evaluation (BE) component is to support administrator in the characterization of PO resources on the basis of benchmarkmeasured performance. Initially, for any relevant benchmark, the administrator submits a JSDL document to the JS component of the GREEN instance associated with his PO. After translating the JSDL document into the particular JSL document compliant with the middleware used by the PO (e.g. JDL for gLite, JDD for Globus), JS passes it to the Benchmark Evaluator port, which interacts with the EE to execute the benchmark against all resources/machines alive. When results are returned, an XML fragment, similar to the one reported in Listing 1, is created for each resource and inserted in a XML document (i.e. Benchmark image), which collects all benchmark evaluations for the PO. The Resource Discovery (RD) is in charge of feeding GREEN with the state of Grid resources. RD operates both locally and globally by carrying out two tasks: 1) to discover the state of the PO resources, 2) to dispatch requests to other GREEN instances. As to the first task, RD dialogues with the underlying IS (e.g. MDS, gLite IS) that periodically reports the state of the PO in the form of an XML file largely conformed to the Glue version adopted by the underlying middleware. This document (namely the PO snapshot) is stored, as it is, in memory and managed by GREEN to answer to external queries issued by other clients (e.g. other GREEN instances, meta-schedulers). To accomplish the dispatching task, RD handles the so called neighbours view, establishing network routes to other nodes. Depending of the number of POs, this view may be limited to a reduced set of network addresses to be contacted individually (as in the case of Figure 3), or deployed via complex data structures and algorithms like those used in Super-Peer networks such as DHT [30] or random walk [31]. The Matchmaker performs the core feature of GREEN: the matching of resources in the Grid and their subsequent ranking, according to the benchmark preferences expressed by the users. Acting as a distributed matchmaker, GREEN manages and compares the benchmark-enriched view of resources with user-submitted jobs, and produces a list of feasible resources (see Figure 4). The task of selecting the “best” among this list, is left to a (meta)scheduler to which the resource set is passed, so allowing to apply the preferred scheduling policies to optimize Grid throughput or other target functions (e.g. response times, QoS,….). Once the “best” resource is chosen, GREEN will be re-invoked to carry-out the submission of the job on it, via the EE. To carry out the exchange of message with other GREEN instances, MM leverages on the services of RD.

Figure 4 exemplifies the submission of an extended JSDL document (i.e. including benchmark requirements) by a user via Grid portal (Step 1). The Resource Selector (RS) forwards the document to the JS component of a randomly selected GREEN instance (2) (e.g. PO1). JS activates the Matchmaker (MM) (3), which, through RD forwards the document to all the other known GREEN instances and contemporaneously checks its local memory (4). All the matchmakers filter their PO snapshot selecting the set of PO resources satisfying the query (including benchmark preferences). The

A Performance-Based Methodology to Improve Grid Exploitation

53

resources identifiers and their corresponding benchmark values are included in a list, called PO list, which is returned back to MM, following the routes expressed by their neighbors’ views (5). MM merges these lists with its own PO list and produce a Global List, ordered on the ranking values, that is passed to JS (7), which returns it back to RS (8). RS applies its scheduling policy to determine the resource to use, and calls the JS of the GREEN responsible of the PO owning the selected machine (GREEN PO2’s instance in our case), by sending it the extended JSDL document along with the data indentifying the selected resource (9). This JS translates the information regarding the job execution of the original JSDL document in the format proper of the specific PO middleware, stating the resource on which the computation takes place (producing a JDD document for GT4 resources or a JDL document for the gLite ones), and finally, activates the Execution Environment in charge of executing the job represented in the translated document (10).

Fig. 4. A user submitting an extended JSDL document via Grid portal

5 Concluding Remarks To satisfactorily fulfil all the potentialities offered by Grids, users have to be supplied with practices and tools, able to overcome the difficulties and obstacles present in

54

A. Clematis et al.

such rich but complex environments. In particular, distributed resources and users applications have to be orchestrated in such a way that user’s objectives are addressed in the most seamless and effective way. We designed GREEN a management tool, primarily devoted to the matching of resources and jobs. It operates at intermediate level between users and Grid middleware, and in this way enables a simplified management of Grid resources. GREEN is based on a benchmarking methodology aimed to evaluate the performance of resources, and allowing users to express her performance preference, through an appropriate extension to Grid submission and description languages such as JSDL and Glue. The appropriateness of our methodological approach is documented by the presentation of some experimental results, which confirmed, in our opinion, the choice of adopting a double level of benchmark, as a means to reduce the gap between users’ needs and resources offer.

References 1. Foster, I., Kesselman, C.: The Grid 2: Blueprint for a New Computing, 2nd edn. Morgan Kaufmann, San Francisco (2003) 2. The cancer Biomedical Informatics Grid Homepage, https://cabig.nci.nih.gov/ 3. The Worldwide LHC Computing Grid Homepage, http://lcg.web.cern.ch/LCG/ 4. The AstroGrid Homepage, http://www.astroGrid.org/ 5. The Business Experiments in Grid, http://www.beinGrid.eu/ 6. Foster, I., Kesselman, C., Tuecke, S.: The Anatomy of the Grid: Enabling Scalable Virtual Organizations. International Journal of Supercomputer Applications, V 15(3), 200–222 (2001) 7. Yu, J., Buyya, R., Ramamohanarao, K.: Workflow Scheduling Algorithms for Grid Computing. In: Metaheuristics for Scheduling in Distributed Computing Environments. Springer, Heidelberg (2008) 8. Bai, X., Yu, H., Ji, Y., Marinescu, D.C.: Resource matching and a matchmaking service for an intelligent Grid. Int. Journal of Computational Intelligence 1(3), 163–171 (2004) 9. Tsouloupas, G., Dikaiakos, M.D.: A Tool for the Interactive Performance Exploration of Grid Infrastructures. Journal of Parallel and Distributed Computing 67, 1029–1045 (2007) 10. Hockney, R.W.: The science of computer benchmarking. Software, environments, tools. SIAM, Philadelphia (1996) 11. Nadeem, F., Prodan, R., Fahringer, T., Iosup, A.: Benchmarking Grid Applications for Performance and Scalability Predictions. In: CoreGrid Workshop on Middleware. Springer, Dresden (2007) 12. Dikaiakos, M.D.: Grid benchmarking: vision, challenges, and current status. Concurrency and Computation - Practice & Experience 19(1), 89–105 (2007) 13. Huedo, E., Montero, R.S., Llorente, I.M.: A Framework for Adaptive Scheduling and Execution on Grids. Software - Practice & Experience 34(7), 631–651 (2004) 14. gLite 3.1 User Guide, Doc. CERN-LCG-GDEIS-722398,(April 28, 2009), https://edms.cern.ch/file/722398/1.2/gLite-3-UserGuide.html 15. Chun, G., Dail, H., Casanova, H., Snavely, A.: Benchmark probes for Grid assessment. In: 18th International Parallel and Distributed Processing Symposium (IPDPS 2004), Santa Fe, New, Mexico, USA. IEEE Computer Society, Los Alamitos (2004)

A Performance-Based Methodology to Improve Grid Exploitation

55

16. Frumking, M., Van der Wijngaart, R.F.: NAS Grid Benchmarks: A tool for Grid space exploration. Cluster Computing 5(3), 315–324 (2002) 17. Elmroth, E., Tordsson, J.: Grid resource brokering algorithms enabling advance reservations and resource selection based on performance predictions. Future Generation Computer Systems 24(6), 585–593 (2008) 18. The Flop Benchmark: http://www.netlib.org/performance/papers/flops/flops_2/ 19. The STREAM Benchmark: Computer Memory Bandwidth, http://www.streambench.org 20. Cachebench Home Page, http://icl.cs.utk.edu/projects/llcbench/cachebench.html 21. MPPTest - Measuring MPI Performance, http://www-unix.mcs.anl.gov/mpi/mpptest/ 22. Bonnie Home Page, http://www.textuality.com/bonnie/ 23. The LINPACK Benchmark: http://www.netlib.org/benchmark/hpl/ 24. Clematis, A., Corana, A., D’Agostino, D., Gianuzzi, V., Merlo, A., Quarati, A.: A distributed approach for structured resource discovery on Grid. In: Proceeding of CISIS 2008, pp. 117–125. IEEE Computer Society, Los Alamitos (2008) 25. Andreozzi, S.: GLUE Specification v. 2.0, rev. 3 (2009), http://forge.Gridforum.org/sf/docman/do/downloadDocument/pro jects.glue-wg/docman.root.drafts/doc15023 26. GLUE v. 2.0 – Reference Realizations to Concrete Data Models (2008), http://forge.Gridforum.org/sf/go/doc15221?nav=1 27. http://www.globus.org/toolkit/docs/4.2/4.2.0/user/gtuser-execution.html 28. Job Description Language Attributes Specification for the gLite Middleware, Doc. EGEEJRA1-TEC-555796-JDL-Attributes-v0-8 (3/5/2006) 29. Anjomshoaa, A., Brisard, F., Drescher, M., Fellows, D., Ly, A., McGough, S., Pulsipher, D., Savva, A.: Job Submission Description Language (JSDL) Specification v1.0. Grid Forum Document GFD 56 (2005) 30. Cai, M., Frank, M., Chen, J., Szekely, P.: A multiattribute addressable network for Grid Information Services. In: Proc. 4th Int. Workshop on Grid Computing (2003) 31. Rabat, C., Bui, A., Flauzac, O.: A Random Walk Topology Management Solution for Grid. In: Bui, A., Bui, M., Böhme, T., Unger, H. (eds.) IICS 2005. LNCS, vol. 3908, pp. 91–104. Springer, Heidelberg (2006)

One Is Not Enough: A Hybrid Approach for IT Change Planning Sebastian Hagen1 , Nigel Edwards2, Lawrence Wilcock2 , Johannes Kirschnick2, and Jerry Rolia2 1

Munich Technical University, Boltzmannstr. 3, 85748 Garching, Germany [email protected] 2 HP Labs, Long Down Avenue, Bristol BS34 8QZ, United Kingdom [email protected]

Abstract. We propose a novel hybrid planning approach for the automated generation of IT change plans. The algorithm addresses an abstraction mismatch between reﬁnement of tasks and reasoning about the lifecycle and state-constraints of domain objects. To the best of our knowledge, it is the ﬁrst approach to address this abstraction mismatch for IT Change Management and to be based on Artiﬁcial Intelligence planning techniques. This has several advantages over previously existing research including increased readability, expressiveness, and maintainability of the descriptions. We developed the foundations of the approach and successfully validated it by applying it to change request planning for TikiWiki, a Content Management System. Keywords: IT Change Management, change planning, policy reﬁnement, AI planning.

1

Introduction

Due to the proliferation of the Software as a Service and the Cloud Computing paradigm, data centers are rapidly growing in size. The complexity of hosted applications increases as well because it has become feasible to host massively distributed applications. This puts additional burden on data center operators and their customers because Change Management becomes more diﬃcult. The generation of change plans is an important step of Change Management as deﬁned by ITIL [7]. Automating the generation of change plans is the key to reduce staﬀ costs, to cope with the complexity of data centers and applications, to comply with company wide policies, and to reduce operator failures. We know of no commercial Data Center Automation product or published research addressing the automated generation of change plans based on workﬂow descriptions, hierarchical reﬁnement strategies, lifecycle behavior and state-constraints of domain objects. Our work addresses an abstraction mismatch inherent to these descriptions. Planning for change requests (CRs) involves the usage of two contrary abstractions: C. Bartolini and L.P. Gaspary (Eds.): DSOM 2009, LNCS 5841, pp. 56–70, 2009. c IFIP International Federation for Information Processing 2009

One Is Not Enough: A Hybrid Approach for IT Change Planning

57

First, reasoning about the state, lifecycle behavior, and state constraints respectively dependencies among domain objects is necessary. For example, a database can be in states like running or stopped. Dependencies refer to the state of other domain objects. For instance, a database can only be installed if the virtual machine is in state on. When reasoning about the behavior of domain objects, the notions of state, lifecycle behavior, and state-constraints of domain objects cannot be avoided. Second, IT change request planning involves the speciﬁcation of best practice workflows and the refinement of abstract high-level CRs into ﬁner grained CRs until non-decomposable CRs are reached. This is necessary when planning for abstract high-level CRs. For example, the task to test an application may be decomposed into a set of subtests. This cannot be expressed by lifecycle states of domain objects and constraints among them - an abstraction mismatch. While this abstraction mismatch has been noted before in the area of Policy Based Management [12], [13], [14], and [11], it has not yet been addressed for IT Change Management. We propose an approach to address this problem for IT change planning. The hybrid approach supports hierarchical task reﬁnement and reasoning about the lifecycle and dependencies of domain objects interchangeably. Its KBs clearly separate hierarchical problem solving strategies, description of domain object behavior, and state-constraints from each other. Thus, domain descriptions produced by an IT practitioner become more readable, extendable, and maintainable because the abstractions are clearly separated and made explicit. The separation also paves the way for a simple methodology to write KBs. The remainder of this paper is organized as follows: Section 2 introduces the basic terms underlying the hybrid approach. The contributions are highlighted in Sect. 3. Section 4 introduces the algorithm based on a TikiWiki planning example. The performance of our prototype is evaluated in Sect. 5. Related Work concerning IT change planning and Policy Based Management is discussed in Sect. 6. Finally, Sect. 7 concludes the paper.

2

Conceptual Model and TikiWiki Planning Domain

This section describes the three levels comprising the conceptual model of our approach. The Domain Entity level in Subsect. 2.1 provides an object oriented model to reason over. The Behavioral level describes the behavior of domain objects. It is introduced in Subsect. 2.2. Finally, we explain the Reﬁnement level which describes workﬂows and task decomposition rules. 2.1

Domain Entity Level

The Domain Entity level deﬁnes two models, the Domain Object Model (DOM) and the Domain CR Model. The Domain Object Model provides an object oriented model representing the infrastructure and hosted software components. An instance of the DOM is used during planning to evaluate preconditions and to make eﬀects of planned operations persistent. The DOM can be based on

58

S. Hagen et al.

modeling techniques like CIM or EMF. We are using Groovy [6] because it is Java compatible, supports conﬁguration templates to instantiate OO models, and Domain Speciﬁc Languages (DSL) can be easily written. In the DOM used throughout this work, a TikiWiki cloud service consists of a database, several Apache servers, and one load balancer. It is shown in Fig. 2. Groovy methods implemented in the DOM can be called by the planner to implement the eﬀects of actions. The DSL describing the planning domain becomes more readable when complex change behavior can be hidden behind method calls. This feature is not supported by traditional Artiﬁcial Intelligence (AI) planners based on predicate KBs, e. g., SHOP2 [9]. This also makes it easier to use our approach with legacy models. We assume that the state of a domain object is automatically kept updated by the DOM’s methods. Note that the DOM can be easily extended to take physical infrastructure into account.

Fig. 1. eSTS for domain objects

Fig. 2. DOM of TikiWiki domain

The Domain CR Model is a model for the diﬀerent kind of CRs. A change request cr is a 6-tuple cr = (ncr , ocr , paramscr , HBcr , parentcr , tcr ), where ncr is the name of cr, ocr the target of the CR, i. e., an instance of class DomainObject in the DOM, aﬀected by cr, paramscr a map of parameters customizing cr, and HBcr a set of CRs happening before cr. Throughout this work HBcr is considered to be non-transitive. The HB sets of all CRs can be used to compute the transitive closure of all CRs happening before a particular CR. parentcr is the parent of cr. tcr ∈ {na, at, sc} denotes the type of a CR with the following characteristics: A non-atomic (na) CR is subject to further reﬁnement into child CRs by a method deﬁned in the Reﬁnement level. An atomic (at) change request cannot be further reﬁned. It has eﬀects on the DOM described by operators in the Reﬁnement level. A state-changing (sc) CR is subject to further reﬁnement; All of it descendants contribute to the state change in domain object ocr , e. g., to resolve dependencies or to actually perform its state-change. 2.2

Behavioral Level

The Behavioral level describes the lifecycle-behavior of domain objects by means of extended restricted state-transition systems (eSTSs), an extended version of restricted STSs [4]. To keep matters simple in our modeled domain, we assume that databases, Apache instances, and load balancers have the same eSTS. A simpliﬁed graphical representation of it is given in Fig. 1. More formally, eSTS σ is a 3-tuple σ = (pσ , Sσ , Tσ ) such that pσ is a precondition evaluated over

One Is Not Enough: A Hybrid Approach for IT Change Planning

59

a domain object o to decide whether σ describes the behavior of o. See the boolean expression in Line 1, List. 1 which associates the eSTS to every instance of class DomainObject. Sσ is the set of states (see Line 2 in List. 1) and Tσ the set of transitions of σ (see Lines 3–13). A transition t ∈ Tσ is a 5-tuple t = (st , gt , crt , Dt pre , Dt post ) such that st ∈ Sσ is the source and gt ∈ Sσ the sink of transition t. Line 4 in Listing 1 deﬁnes the stop transition from the Started to the Installed state. crt is the CR that needs to be achieved to change the state from st to gt . For example, Line 5 in List. 1 links a CR called stop, i. e., ncr == stop, to transition stop. Dt pre and Dt post are dependencies that need to be fulﬁlled before or after crt is planned for, i. e., before or after the transition is taken in the eSTS. Dt pre = (pt pre , CRSt pre ) where pt pre is a precondition evaluated over a domain object to determine whether the dependency CRs described in CRSt pre need to be planned for before crt . For instance, the precondition of the dependency given in Lines 7–12 in List. 1 says that it is only valid for an instance of Database. The dependency CRs (CRSt pre ) of the dependency are deﬁned in Lines 11–12. They describe state-changing CRs to stop all Apache servers, i. e., to change their state to Installed. Thus, the dependency CRs are only planned for if the stop transition in a database is taken. If the stop transition in an Apache server is taken, then the depedency does not apply because the precondition evaluates to false. All in all, the DSL given in Listing 1 describes the eSTS linked to all domain objects, its three states, the stop transition, the stop CR linked to the transition, and a dependency to stop all Apache instances before the stop transition is executed in a database. The other transitions and dependencies were omitted due to space constraints. Listing 1. DSL for eSTS 1 2 3 4 5 6 7 8 9 10 11 12 13 14

eSTS ( i t . i n s t a n c e o f DomainObject ) { s t a t e s { [ ” Removed ” , ” I n s t a l l e d ” , ” S t a r t e d ” ] } transitions { t r a n s i t i o n ( ” s t o p ” , from : ” S t a r t e d ” , t o : ” I n s t a l l e d ” ) { subtask { stop t a r g e t : i t . t a r g e t } dependencies { d e p e n d e nc y{ p r e c o n d i t i o n { i t . t a r g e t i n s t a n c e o f D a t a b a se} t y p e { t i m e : ” b e f o r e ” , decomp : ” p a r a l l e l ”} subtasks { i t . t a r g e t . A p a c h e i n s t a n c e s . each{ s e t −s t a t e t a r g e t : i t , g o a l s t a t e : ” I n s t a l l e d ” }}}...}}...} }

Note that the dependency in List. 1 only applies for a database (Line 8) while a database has the same states and transitions as every other domain object. The DSL decouples the dependency speciﬁcation from the lifecycle speciﬁcation to specify domain objects with the same lifecycle but with diﬀerent dependency behavior within one eSTS. This increases the usability and reuseability of the Knowledge Base. The code in Line 11 in List. 1 directly navigates the DOM instance to create the dependency CRs (CRSt pre ) of transition stop. it refers to the state-changing CR which the planner plans for using the eSTS the dependency

60

S. Hagen et al.

is speciﬁed within. By following the target reference the database domain object (ocr ) of the DOM instance is reached holding a list of all Apache instances (see Apache instances reference in Fig. 2). We only need to specify dependencies that directly aﬀect other domain objects when executing a transition. For example, the eSTS holds a before-dependency to stop the Apache servers before the database is stopped (Lines 7–13). The eSTS describing Apache, in this case the same eSTS, holds a before-dependency for Apache instances to stop the load balancer before transition stop is taken. This is how we can exploit transitivity over dependencies when stopping a database. Note that the latter dependency is only valid if the Apache to stop is the last one running in the system. Such constraints can be deﬁned in the precondition (pt pre or pt post ) of a dependency. 2.3

Refinement Level

The Refinement level describes rules for CR decomposition, workﬂows, and best practice problem solving strategies not expressible by eSTSs and dependencies. A (Hierarchical Task Network (HTN)) method m [3], [4] describes a sequential or parallel decomposition of a non-atomic CR into child CRs. More formally let m = (nm , pm , CRSm ) be a 3-tuple where nm is the name of m, pm is a precondition to determine whether m is applicable, CRSm is a set of child CRs to be achieved in order to achieve the CR m is applied to. The children can either be in no temporal relationship, i. e., ∀cr ∈ CRSm : HBcr = ∅, or they can be in a sequential relationship such that HBcrn+1 = {crn }. Let cr be a non-atomic CR, i. e., cr = (ncr , ocr , paramscr , HBcr , parentcr , na), then m can be applied to decompose cr iﬀ ncr == nm and pm is satisﬁed by ocr , i. e., their names match and the precondition evaluates to true over the target of cr. More practically, patching a domain object is deﬁned by the following workﬂow: (CR1) The domain object to patch is stopped. (CR2) The update is applied. (CR3) The domain object is tested. (CR4) The previous state of the patched domain object is restored. The decomposition of patch incorporates state-changing CRs (1 and 4) and non-atomic/atomic CRs (2 and 3). It is tasks like patching and testing where the hybrid approach can play out its advantages because planning works interchangeably between reﬁnement and state-transition systems. The KB engineer can rely on the previously deﬁned behavior and dependencies in the Behavioral level when writing methods. Listing 2. DSL for patch method 1 2 3 4 5 6 7 8

method ( name : ” p a t c h ” , p r e : i t subtasks { sequential { String state old = it . target s e t −s t a t e t a r g e t : i t . t a r g e t update t a r g e t : i t . t a r g e t test target : it . target s e t −s t a t e t a r g e t : i t . t a r g e t

. target

i n s t a n c e o f DomainObject ) {

. state , goal state

: ” Installed ”

,

:

goal state

s t a t e o l d }}}

One Is Not Enough: A Hybrid Approach for IT Change Planning

61

Listing 2 shows the DSL for the patch method. Line 1 describes the name nm (patch) and the precondition pm (target of CR, i. e., ocr , is instance of class DomainObject). Lines 2–8 describe CRSm as the sequential decomposition of the patch CR into CRs 1-4 as Groovy code. Line 4 saves the current state of ocr in the local variable state old. Lines 5 and 8 describe state-changing CRs to stop (CR1) and to restore (CR4) the previous state of the target of the patch CR (ocr ). The previously locally deﬁned variable state old is used as the goal state parameter of the last state-changing CR to restore the domain object’s old state in Line 8. Intermediate CRs with names update and test need to be planned for. update is an atomic CR, i. e., a non-decomposable CR with eﬀects on the DOM. Atomic CRs are implemented by (HTN) operators [3], [4] also speciﬁed in the Reﬁnement level. More formally, an operator o is a 3-tuple o = (no , po , eo ). no is the name of o. See Line 1, Listing 3 for an operator named update. po is a precondition to determine whether o can be applied and eo describes the eﬀects of o. Let cr = (ncr , ocr , paramscr , HBcr , parentcr , at) be an atomic CR. o is applicable to cr if ncr == no and po is satisﬁed by ocr . In case of update the precondition po is speciﬁed in Lines 1–2 in Listing 3. It demands that ocr is an instance of class DomainObject and currently in state Installed. The eﬀects eo are deﬁned by the programmatic Groovy code in Lines 4–5. The code is executed by the planner, increasing the version attribute of ocr (Line 4) and returning true to signal the successful execution to the planner. Note that exceptions from the DOM instance can be caught to return false to the planner to trigger backtracking. A method to reﬁne test into subtests is not given because with patch an exemplary method has already been provided. Listing 3. DSL for update operator 1 2 3 4 5 6

o p e r a t o r ( name : ” u p d a t e ” , p r e : i t . t a r g e t && i t . t a r g e t . s t a t e == ” I n s t a l l e d ” ) { effects { i t . t a r g e t . v e r s i o n++ return true }}

i n s t a n c e o f DomainObject

Having deﬁned the Behavioral level previously, there is no need to worry about dependencies and the behavior of domain objects. Behavior described in eSTSs (see List. 1) is strictly separated from reﬁnement strategies described by methods (see List. 2) and operators in List. 3. Both abstractions can be written by diﬀerent KB engineers. One an expert in the lifecycle-management of applications and the the other one an expert in best practices in Change Management workﬂows. Linking Behavioral and Reﬁnement level together overcomes the abstraction mismatch enabling IT practitioners to more naturally express the planning domain. The clear separation between Reﬁnement and Behavioral level makes it easier to write and change domain descriptions. For example, to extend the behavior of a domain object states and transitions can be added. The reﬁnement strategies stay unchanged. Vice versa, the same applies when changing workﬂows.

62

3

S. Hagen et al.

Contributions of the Hybrid Approach

Task reﬁnement and state-based reasoning with constraints are important for IT change request planning. The hybrid approach brings both abstractions together. Previous work regarding IT change request planning has been focused on either of the two abstractions. Only supporting Reﬁnement as done by HTN Planners [3], [9] is powerful enough to plan for CRs, but has several drawbacks. Without a Behavioral level, eSTSs have to be described by methods in the Reﬁnement level. For instance, the eSTS shown in Fig. 1 can be in 3 states and any statechanging CR regarding this eSTS can have two goal states diﬀerent from the current state. Thus, 3 ∗ 2 = 6 methods need to be written as a replacement for state-changing CRs. In general, n ∗ (n − 1) methods have to be written for a domain object with n states. These ’state-changing’ methods will be mixed with higher-level workﬂow methods like the patch method. Furthermore, additional HTN methods are needed to describe diﬀerent dependency behavior for domain objects with the same lifecycle. All in all, the KB becomes more diﬃcult to read and to maintain because concepts are mixed. Other reﬁnement approaches [1], [2] restrict reﬁnement to be based on dependencies. Workﬂows not driven by dependencies, e. g., restarting an application, are diﬃcult to describe in such an approach. Only providing a Behavioral level [5] and no reﬁnement capabilities reduces expressiveness. For instance, the patch and test CRs cannot be planned for only based on state-constraints. There is no convenient way to express that subtests need to be done by using eSTSs and state-constraints. It is more natural to express this as a reﬁnement rule. Even if reﬁnement could be described using eSTSs, dependencies, or state-constraints, the drawback that reﬁnement is mixed with eSTSs or constraints still exists. The hybrid approach oﬀers native support for both abstractions, clearly distinguishes between them, and keeps them separated in the Knowledge Base. However, reasoning about reﬁnement and states can naturally refer to each other. This oﬀers two advantages when comparing it to previous work in IT change planning. First, to the best of our knowledge it is the ﬁrst approach to natively support workflow-, task-refinement-, and statebased-planning based on constraints at the same time. Second, KBs of the hybrid approach are easier to maintain and to read because the abstractions are described in their most natural way and are separated from each other. To the best of our knowledge, it is also the ﬁrst work to apply ﬁndings from AI planning to IT change planning. HTN planning [3], [4] and our extensions to plan for state-changing CRs reason about the eﬀect a CR has on the DOM. Plans are sound from a computational point of view if the algorithm, methods, operators, and eSTSs are sound. We leave this proof to future work. We also show that planning can be done with reasonable overhead over an OO model. This bridges the gap between AI planning and IT Change Management because models like CIM have been traditionally used to represent the IT system to manage.

One Is Not Enough: A Hybrid Approach for IT Change Planning

4

63

The Algorithm

The algorithm takes a 4-tuple (M, O, , Q) as input where M is a set of methods, O a set of operators, a set of eSTSs, and Q a queue of CRs to plan for. To plan for the patch CR, Q holds the ordered set {cr1 } where cr1 = (patch, db, [:], ∅, null, na) at the initial call to the algorithm. See Fig. 3 for the decomposition tree with root node cr1 and all of its descendants created during planning. Note that ocr1 == db, i. e., the database is to be patched. The underlying DOM instance consists of one database (db), one Apache server (ap), and one load balancer (lb). All of them are in state Started.

Fig. 3. Decomposition tree for patch CR

4.1

Decomposing Non-atomic CRs

Algorithm 1 describes the decomposition of non-atomic CRs, e. g., cr1 . A set of applicable methods to decompose cr1 is determined (Line 2) as described in Subsect. 2.3. Only one method m, the method previously provided in List. 2, is applicable. It is chosen in Line 3. CRSm , the list of child CRs described by m, is {cr2 , cr8 , cr9 , cr10 }. Note that each cr of these has the preceding sibling in its HBcr set to formalize the sequential relationship among the children. See the patch method in List. 2 and the tree in Fig. 3 for temporal constraints. For each descendant the parent is set to cr1 in Line 5. Finally, cr1 is removed from the queue Q and CRSm are added at the front, leading to Q = {cr2 , cr8 , cr9 , cr10 }. 4.2

Decomposing State-Changing CRs

cr2 is extracted from Q and passed to Algorithm 3. Line 2 determines a matching eSTS for db, the target (ocr2 ) of cr2 . See Subsect. 2.2 for the matching process. We assume that there always exists exactly one eSTS σ mapped to a domain object o. The goal state (Installed) to be achieved in σ is extracted from the parameters of cr2 and c is initialized with the current state of db (Started) in Line 3. T holds the path from c to g (Line 4). We assume that the path between every state-pair is unambiguous. The path only consists of transition stop, thus T = {t} where t = (Started, Installed, stop, Dt pre , Dt post ). Some helper variables

64 1 2 3 4 5 6 7 8

S. Hagen et al. if cr == (ncr , ocr , paramscr , HBcr , parentcr , na) then Be M = {(nm , pm , CRSm ) ∈ M |pm is satisﬁed over ocr ∧ nm == ncr } Choose non-deterministically m ∈ M foreach cr ∈ CRSm do parentcr = cr end Q = CRSm ◦ rest(Q) end

Algorithm 1. Planning for non-atomic CRs

1 2 3 4 5

if cr == (ncr , ocr , paramscr , HBcr , parentcr , at) then Be O = {(no , po , eo ) ∈ O|po is satisﬁed over ocr ∧ no == ncr } Choose non-deterministically o ∈ O if eﬀects eo are succesfully applied to ocr then Q = rest(Q) else backtrack() end

Algorithm 2. Planning for atomic CRs

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24

if cr == (ncr , ocr , params cr , HBcr , parentcr , sc) then Be σ = (pσ , Sσ , Tσ ) ∈ a eSTS such that pσ is satisﬁed over ocr Let g = paramscr [goal state]; Let c be the current state of ocr Let T ⊆ Tσ be an ordered set of transitions leading from c to g in σ crt old = null; CRSpost old = null; children = [] foreach t == (st , gt , crt , (pt pre , CRSt pre ), (pt post , CRSt post )) ∈ T do if pt pre evaluates to f alse over ocr then CRSt pre = ∅ if pt post evaluates to f alse over ocr then CRSt post = ∅ children = children ◦ CRSt pre ◦ [crt ] ◦ CRSt post HBcrt = HBcrt ∪ CRSt pre ∀cr ∈ CRSt post : HBcr = HBcr ∪ [crt ] if crt old = null ∧ CRSpost old = null then switch (CRSpost old , CRSt pre ) do case (∅, ∅) : HBcrt = HBcrt ∪ [crt old ] case (∅, = ∅) : ∀cr ∈ CRSt pre : HBcr = HBcr ∪ [crt old ] case ( = ∅, ∅) : HBcrt = HBcrt ∪ CRSpost old case ( = ∅, = ∅) : ∀cr ∈ CRSt pre : HBcr = HBcr ∪ CRSpost old end end crt old = crt ; CRSpost old = CRSt post end ∀cr ∈ children : parentcr = cr; Q = children ◦ rest(Q) end

Algorithm 3. Planning for state-changing CRs

One Is Not Enough: A Hybrid Approach for IT Change Planning

65

are initialized in Line 5. After that, Lines 6–22 iterate over the transitions in path T . pt pre evaluates to true over db in Line 7. pt pre is deﬁned in Line 8 in List. 1. The precondition for the dependency applies because t is taken within a database. Thus, CRSt pre == {cr3 } stays untouched. If pt pre did not apply, the dependency tasks in CRSt pre are erased because they do not become children (Line 9). The same is done regarding post-dependencies of t. There is no postdependency, thus pt post is false. Line 9 builds the children comprising transition t. In our case children = []◦[cr3 ]◦[cr7 ]◦[] = [cr3 , cr7 ]. The HB sets of the children need to be adapted. All pre-dependency CRs, i. e., the CRs in CRSt pre , need to happen before crt (cr7 ) (Line 10). In our case HBcr7 = ∅∪{cr3 } = {cr3 }. Line 11 adapts the HB relation in every post-dependency CR because crt happens before them. Nothing is done because CRSt post == ∅. The if statement in Lines 12–20 checks whether t is at least the second transition taken in T . As T only consists of t the body is not executed. For two consecutive transitions t1 ∈ T and t2 ∈ T , the body of the if statement is executed to adapt the HB sets of CRs in crt1 , CRSt1 post , CRSt2 pre , and crt2 to reﬂect that CRs of t2 need to happen after CRs of t1 . Four cases, depending on the values of CRSpost old , the post-dependencies of the previous transition, and CRSt pre , the pre-dependencies of the current transition, are distinguished using pattern matching in the switch statement in Lines 13–19. (1) Both sets are empty. This means crt1 and crt2 directly follow each other. Thus, crt old , storing the task linked to the previous transition, is added to HBcrt (Line 14). (2) The previous transition did not have postdependencies and there are pre-dependencies for the current transition. In this case, the task linked to the previous transition (crt old ) needs to happen before each cr ∈ CRSt pre . (3) CRSpost old = ∅ and CRSt pre == ∅, i. e., crt directly follows the CRs in CRSpost old . Every CR in CRSpost old needs to happen before crt . Thus, CRSpost old is added to HBcrt in Line 16. (4) Both sets are non-empty. Every post-dependency CR of the previous transition needs to happen before every pre-dependency CR of the current transition (Line 18). Line 20 saves crt in crt old and the post-dependencies of t in CRSpost old to refer to them in the next iteration when setting dependencies. Finally, the parent of the children is set to cr and they are added at the front of the queue in Line 23. Q now holds {cr3 , cr7 , cr8 , cr9 , cr10 }. Planning continues with cr3 and cr4 which are similarly decomposed as cr2 . 4.3

Planning for Atomic CRs

cr5 is the ﬁrst atomic CR to be planned for. Algorithm 2 determines all applicable operators for cr5 in Line 2. Applicability of operators is deﬁned in Subsect. 2.3. We assume that there exists an applicable operator for the stop CR. It’s eﬀects are applied to ocr5 , lb, in Line 4. For example, the operator could call a stop method on lb. If the operator returns true, cr5 is removed from Q and planning continues with cr6 and cr7 (see Fig. 3) applying the same operator to them. If the execution of the operator failed, i. e., it returns false, backtracking is triggered. Backtracking ﬁnds a previously planned CR, that has another decomposition alternative and restarts

66

S. Hagen et al.

planning from that CR onwards to generate a non-failing decomposition. The next CR in Q to plan for is cr8 , the atomic update CR. The operator previously given in List. 3 can be applied because its precondition is satisﬁed by db. The operator increases the version attribute of db. Planning continues with cr9 the non-atomic CR to execute the tests. We do not further elaborate on its decomposition because we already showed how non-atomic CRs are decomposed based on cr1 . cr10 , a state-changing CR to start db, is next in Q. As cr2 left db in state Installed, only transition start (t) needs to be taken. There are no depedencies associated to t, thus cr10 is only decomposed into crt , an atomic start task (cr11 ). Planning ends with the application of an operator to cr11 . 4.4

Handling Conflicts among CRs

CRs are always planned for in sequential order. Two CRs happening in parallel could target the same domain object leading to unsound plans if their eﬀects interfere and an execution engine does not execute them the order they were planned for. An additional temporal constraint needs to be added to keep the plan sound. Let HBcr be the set of all CRs planned and happening before cr. parentcr denotes the parent of cr, descendantscr the set of all descendants of cr, and HBcr the local happening before relation of cr. HBcr , the multiset of all CRs planned and happening before cr, is deﬁned as follows: HBcr = HBcr ∪ cr ∈HBcr descendants(cr ) ∪ HBparent ∪ cr ∈HBcr HBcr . cr Less formally, HBcr consists ofall CRs directly happening before cr (HBcr ), all descendants of these CRs ( cr ∈HBcr descendants(cr )), all CRs beeing planned for and happening before the parent HBparent , and of all CRs transicr tively happening before cr ( cr ∈HBcr HBcr ). Let Ωcr be the set of CRs already part of the decomposition tree before planning for cr. Let anchestorscr be the set of all anchestors of cr. Then HPcr , the set of all CRs previously planned for and happening in parallel to cr, is deﬁned as HPcr = Ωcr − {HBcr ∪ anchestorscr }. Using HPcr rules can be deﬁned to resolve conﬂicts between parallel CRs. Two constraints need to be checked before planning for a CR cr:

– If there exists a cr ∈ HPcr with the same target, i. e., ocr == ocr , then cr needs to happen after cr , i. e., HBcr = HBcr ∪ [cr ]. This prevents parallel execution of cr and cr by an execution engine. Our approach assume that CRs targeting diﬀerent domain objects can be executed in parallel. – If cr is a state-changing CR and a state-changing CR cr ∈ anchestorscr exists, such that ocr == ocr , then planning needs to fail because an anchestor CR of cr already tries to change the state of the same domain object.

5

The Prototype

This section introduces and evaluates the performance of the developed prototype. Our experiments were conducted using WinXP, an Intel Xeon with 3Ghz, and 1GB of RAM. The planner ran in non-Gui mode to evaluate the performance

One Is Not Enough: A Hybrid Approach for IT Change Planning

67

of the pure algorithm. We planned for the patch CR using our fully developed KB of the TikiWiki domain. A small underlying DOM instance comprising one database, three Apache servers, and one load balancer produces a plan (the atomic leaf nodes of the decomposition tree) consisting of 64 atomic CRs. In total, the planner had to plan for 153 CRs (30 na, 64 at, and 59 sc) to completely decompose the patch CR. The depth of the decomposition tree is 11. Planning took 3.6 seconds and 1,900 domain objects were serialized. Serialization of domain objects is necessary to restore older instances of the DOM in case the planner backtracks. With increased size of the DOM planning takes longer. Planning for 10 Apache servers results in a decomposition tree with 419 CRs, a plan comprising 176 atomic CRs, and 14,000 domain object serializations. Total planning time is 14.3 seconds, whereas 6.7% are spent on serialization consuming 1.6 MB of main memory. The serialization and deserialization mechanism does not turn out to be a signiﬁcant bottleneck. For instance, serializing 100,000 large domain objects takes 5.5 seconds and consumes 38.5 MB of main memory. Deserialization on backtracking is slightly slower because the target reference has to be rerouted to the restored domain object for each CR of the decomposition tree. The prototype serializes the DOM instance before planning for any CR. Instead, it is suﬃcient to serialize the DOM instance only for atomic CRs because only these can change the model. To restore the model for a non-atomic or state-changing CR the DOM instance associated to the latest planned atomic CR needs to be restored. In case of the patch example with three Apache instances 48% less domain objects are serialized. All in all, the prototype proves that the generation of large plans is possible within reasonable time compared to the actual execution time of the plan.

6

Related Work

This section discusses related work in IT change planning, policy reﬁnement, and AI planning. CHAMPS by Keller et al. [8] formalizes planning and scheduling as an optimization problem achieving a high degree of parallelism. Planning is solely based on dependencies. CHAMPS does not address the abstraction mismatch. Compared to our approach, the algorithm does not reason about the eﬀects of actions. It is diﬃcult to reason about interaction of actions not previously ascertained in a dependency structure. CHAMPS includes scheduling and domain descriptions are automatically derived, while we focus on planning and rely on an IT practitioner to describe the domain. Diﬀerent to CHAMPS, we focus on state-related constraints. However, we successfully planned for nonstate related constraints using methods and suitable preconditions. Cordeiro et al. [1] introduce the notion of templates to reuse knowledge in IT change design. Plan templates can be described by methods in our approach. Diﬀerent to CHAMPS, their algorithm plans for task reﬁnement based on dependencies. Compared to our work, the lifecycle-behavior and state-constraints of domain objects are not made explicit. Furthermore, reﬁnement not based on

68

S. Hagen et al.

dependencies is diﬃcult to describe. Similar to CHAMPS, eﬀects of actions are not taken into account. Aware of this, an algorithm that takes the eﬀects of actions into account is later proposed in [2]. Compared to our solution reﬁnement of tasks is solely based on dependencies, no native support for state-constraints is given, and lifecyclebehavior is not made explicit. In addition to that, interchangeable reasoning about reﬁnement not necessarily based on dependencies and lifecycle-behavior is not readily possible. Thus, Cordeiro et al. do not address the abstraction mismatch. The work focuses on reﬁnement driven by constraints. Goldsack et al. [5] argue in favor of a pure declarative, state-based approach to manage large data centers. Workﬂows are considered harmful due to side eﬀects, concurrent changes, and their procedural instead of declarative nature. Similar to CHAMPS reasoning by reﬁnement is not supported. Diﬀerent to Goldsack et al. our solution supports reﬁnement. Also related to our work is the area of Policy Based Management [10], in particular policy reﬁnement. It is generally concerned with the reﬁnement of abstract high-level Quality of Service (QoS) goals into lower-level policies to achieve the higher-level policy. There are subtle diﬀerences between IT change planning and policy reﬁnement. Traditionally, plans generated in IT Change Management do not describe alternatives and are very much driven by best practices. Diﬀerent to that, change plans in policy reﬁnement are goal driven and tend to consist of on event do something policies that can be even stored for later evaluation. Thus, plans to achieve a high-level policy tend to incorporate diﬀerent alternatives or might be adapted when events occur. Recent work on policy reﬁnement has taken the abstraction mismatch into account. Bandara et al. [12], [13] propose a Goal-oriented policy reﬁnement technique based on the Event Calculus and abductive reasoning techniques. High-level goals formalized in temporal logic are reﬁned until System Goals are reached using formal reﬁnement patterns introduced by Darimont et al. [11]. System Goals can be implemented by state changes (called a strategy). Bandara et al. [12], [13] use abductive reasoning, while Rubino-royal et al. [14] propose to use model checking to derive strategies. Compared to our work, reasoning is proven to be sound but is limited to a small set of goal decomposition patterns [11]. We do not use abductive reasoning [13] or model checking [14] to derive a strategy but state exploration of the STS. In [12], [13], and [14] state changes to domain objects do not trigger further reﬁnement steps. Low-level goals always address a state to be achieved. This is diﬀerent to our approach, where operators do not need to change the lifecycle-state of a domain object. Similar to their work, we use transition systems to describe the behavior of domain objects. Also related to our work is the area of AI planning, particularly HTN planning, which is used in the Reﬁnement level. It has been subject to intense research [4]. It does not natively support to reason about the lifecycle-behavior of domain objects, justifying the research regarding the hybrid approach. To the best of our knowledge HTN has not yet been applied to IT change planning.

One Is Not Enough: A Hybrid Approach for IT Change Planning

7

69

Conclusion and Future Work

We have identiﬁed an abstraction mismatch between task reﬁnement, reasoning about the lifecycle of domain objects, and state-constraints in the area of IT change request planning. We have proposed a hybrid algorithm to natively address these abstractions and to reason interchangeably about them. Our results are quite positive. Having previously written KBs for IT change planning in SHOP2 [9], a pure HTN planner, we found it easier to write and to maintain KBs for the hybrid approach due to the clear separation of concepts. Our prototype proves that the algorithm can generate larger plans within reasonable time and that OO models can be used as KBs to be planned over with reasonable overhead for model backup and restoration. For future work we envision a hybrid algorithm using a declarative speciﬁcation of state constraints and task reﬁnement to address the abstraction mismatch. Additional value could be generated because thinking declaratively about state-constraints is easier than the procedural like approach we explored in this paper. Furthermore, it provides us with more freedom to specify state-changing CRs aﬀecting many domain objects. We envision a Groovy based language inspired by ﬁrst-order-logic to qualify and quantify domain objects of the Domain Entity level that are aﬀected by state constraints. In addition to that, we want to examine scalability, replanning, optimization, and scheduling techniques regarding the hybrid approach.

References 1. da Costa Cordeiro, W.L., Machado, G.S., Daitx, F.F., et al.: A template-based solution to support knowledge reuse in IT change design. In: Proceedings of Network Operations and Management Symposium 2008, pp. 355–362 (2008) 2. da Costa Cordeiro, W.L., Machado, G.S., Andreis, F.G., Santos, A.D., Both, C.B., Gaspary, L.P., Granville, L.Z., Bartolini, C., Trastour, D.: A Runtime ConstraintAware Solution for Automated Reﬁnement of IT Change Plans. In: De Turck, F., Kellerer, W., Kormentzas, G. (eds.) DSOM 2008. LNCS, vol. 5273, pp. 69–82. Springer, Heidelberg (2008) 3. Erol, K., Hendler, J., Nau, D.S.: HTN planning: Complexity and expressivity. In: Proceedings of the Twelfth National Conference on Artiﬁcial Intelligence (AAAI 1994), vol. 2, pp. 1123–1128. AAAI Press/MIT Press, Washington (1994) 4. Ghallab, M., Nau, D.S., Traverso, P.: Automated Planning: Theory & Practice. Morgan Kaufmann, San Francisco (2004) 5. Goldsack, P., Murray, P., Farrell, A., Toft, P.: SmartFrog and Data Centre Automation. In: HPL Technical Report, HPL-2008-35 (2008), http://www.hpl.hp.com/techreports/2008/HPL-2008-35.pdf 6. Groovy Homepage, http://groovy.codehaus.org/ 7. IT Infrastructure Library: ITIL Service Transition (V3), http://www.itil-officialsite.com/home/home.asp 8. Keller, A., Hellerstein, J.L., Wolf, J.L., Wu, K.-L., Krishnan, V.: The CHAMPS system: change management with planning and scheduling. In: Proceedings of Network Operations and Management Symposium (NOMS 2004). IEEE/IFIP, pp. 395–408 (2004)

70

S. Hagen et al.

9. Nau, D., Muoz-avila, H., Cao, Y., Lotem, A., Mitchell, S.: Total-order planning with partially ordered subtasks. In: Proceedings of the Seventeenth International Joint Conference on Artiﬁcial Intelligence (IJCAI 2001), pp. 425–430 (2001) 10. Boutaba, R., Aib, I.: Policy-based Management: A Historical Perspective. Journal of Network and Systems Management, 5(4), 447–480 (2007) 11. Darimont, R., Lamsweerde, A.: Formal reﬁnement patterns for goal-driven requirements elaboration. ACM SIGSOFT Software Engineering Notes 21(6), 179–190 (1996) 12. Bandara, A.K., Lupu, E.C., Moﬀet, J., Russo, A.: A Goal-based Approach to Policy Reﬁnement. In: Proceedings of the Fifth IEEE International Workshop on Policies for Distributed Systems and Networks (POLICY 2004), Washington, DC, USA, p. 229. IEEE Computer Society, Los Alamitos (2004) 13. Bandara, A.K., Lupu, E.C., Russo, A., Dulay, N., Sloman, M., Flegkas, P., Charalambides, M., Pavlou, G.: Policy reﬁnement for DiﬀServ quality of service management. In: Proceedings of 9th IFIP/IEEE International Symposium on Integrated Network Management (IM 2005), pp. 469–482 (2005) 14. Rubio-Loyola, J., Serrat, J., Charalambides, M., Flegkas, P., Pavlou, G., Lafuente, A.L.: Using linear temporal model checking for goal-oriented policy reﬁnement frameworks. In: Proceedings of Sixth IEEE International Workshop on Policies for Distributed Systems and Networks (POLICY 2005), pp. 181–190 (2005)

Improving IT Change Management Processes with Automated Risk Assessment Juliano Araujo Wickboldt1 , Lu´ıs Armando Bianchin1 , Roben Castagna Lunardi1, Fabr´ıcio Girardi Andreis1 , Weverton Luis da Costa Cordeiro1 , Cristiano Bonato Both1 , Lisandro Zambenedetti Granville1 , Luciano Paschoal Gaspary1 , David Trastour2 , and Claudio Bartolini3 1

Federal University of Rio Grande do Sul, Porto Alegre, Brazil 2 Hewlett Packard Laboratories, Bristol, UK 3 Hewlett Packard Laboratories, Palo Alto, USA

Abstract. The rational management of IT infrastructures is a goal of modern organizations that aim to deliver high quality services to their customers in an affordable way. Since changes are imminent in such a dynamic environment, failures during this process may directly affect business continuity. Hence, risk assessment is a key process in IT change management. Despite its importance, risks are usually assessed by humans based on empirical knowledge, leading to inaccurate basis for decision making. In this paper, we present a solution for automating the risk assessment process, which combines historical data from previous changes and analyzes impact of changes over affected elements. A prototypical system was developed to evaluate the solution on an emulated IT infrastructure. The results achieved show how the automated solution is capable of raising the quality of changes, therefore reducing service disruption caused by changes.

1 Introduction Modern organizations take advantage of information technology (IT) resources and services to add value to their businesses. The heterogeneity of these technologies, which together constitute an IT infrastructure, makes the task of IT management increasingly complex. In this scenario, the rational management of IT infrastructures improves the quality of provided services and reduces operational costs. For consistent and secure maintenance of these infrastructures, the Office Government Commerce (OGC) has introduced the Information Technology Infrastructure Library (ITIL) [1], which is a set of processes and best practices that provides guidance for the proper management of IT resources and services. Being one of the core processes of ITIL, change management [2] provides general guidelines for conducting changes over IT infrastructures, from the early specification to the final deployment and evaluation. It defines that all changes should be described in a document called Request for Change (RFC). An RFC specifies, in a declarative way, what should be done and the primary Configuration Items (CIs) affected (devices, applications, services, etc.), but not detailing how the change should be implemented. This must be indeed performed by human operators or even by an automated management system. Subsequently, RFCs must be reviewed, approved, and scheduled by the Change C. Bartolini and L.P. Gaspary (Eds.): DSOM 2009, LNCS 5841, pp. 71–84, 2009. c IFIP International Federation for Information Processing 2009

72

J.A. Wickboldt et al.

Advisory Board (CAB). This committee, usually chaired by a change manager, should be composed of people with extensive knowledge on the organization’s processes, often coming from different areas, but not necessarily familiar with the underlying technologies deployed in IT infrastructure. IT infrastructures support services that are essential for business continuity. Hence, when changes to the managed infrastructure are required, the risks associated with it should be considered. According to ITIL, risks should be measured and treated before a change is approved. Risk mitigation aims to reduce the possibility of changes causing unnecessary disruption to changed services. Risks in IT change management should be observed as a combination of the probability of occurrence of potentially negative events and their impact to business continuity [1]. Examples of such events include: failure on software installation, incorrect configurations, and physical defects in CIs. Risk assessment has been typically performed by human operators, often based only on empirical knowledge. However, due to the large number of CIs associated with a change request and the amount of variables that should be considered (e.g., history of failures and impact of affected CIs), the adoption of such approach may end up presenting superficial and/or inaccurate results to serve as basis for decision making. Despite the recommendations proposed by ITIL, it does not present a practical method for risk assessment in change management. Recently, some authors have proposed solutions for the automation of change management in its several phases [3] [4] [5]. Nevertheless, no previous work proposed an automated approach for the risk assessment in the planning phase of change management. By employing a proper method for risk assessment, an automated system could aid the human operator to quickly identify threats in a requested change before deploying it to the IT infrastructure. Based on risk reports, the operator would be able to modify the original RFC or even adapt the IT infrastructure, in order to reduce the possibility of occurrence of change related incidents. To address the aforementioned issue, we propose in this paper a solution for automating risk assessment in IT change management. Our solution is based on the history of executions of changes over an IT infrastructure, observing the occurrence of previous deployment failures and identifying potential issues for future executions. With this solution we aim to provide a change management system with support for proactive treatment of incidents, enabling operators to redesign changes in order to reduce occurrence deployment failures upon change executions. The remainder of this paper is organized as follows. Section 2 briefly reviews some of the most prominent research initiatives in risk and change management. Section 3 details our solution for automated risk assessment, whereas Section 4 describes the prototypical implementation developed. In Section 5 we present an experimental evaluation conducted to measure the results of the solution. Finally, Section 6 closes this paper with concluding remarks and prospective directions for future work.

2 Related Work Risk management is a cross-discipline that has been investigated and employed in several different areas. Risk assessment, for example, can be a tool for guiding financial investments [6], health care decisions [7], and the strategies of insurance companies

Improving IT Change Management Processes with Automated Risk Assessment

73

[8]. According to the Institute of Risk Management (IRM)1 , the risk management discipline defines the process whereby organizations methodologically address the risks associated with their activities, aiming at achieving sustained benefits [9]. The literature usually defines risks as events whose potential consequences may be either positive or negative to the successful accomplishment of a goal. However, in practice, the negative aspect is far more considered, mainly in critical areas such as health care. The actual result is that risk management becomes strongly focused on the prevention and mitigation of harms. This observation also holds in the investigations on risks associated to the design and operation of computational systems. Some authors have employed probabilistic models to predict undesired events as well as estimate metrics for risk management in IT. Fewster and Mendes [10] have proposed a framework that, using a Generalized Linear Model (GLM), is able to analyze the risks associated with the development of Web-based systems. The authors showed that GLM was effective in predicting the risks of, for example, overcoming project budget or violating final deployment deadlines. Hearty et al. [11], in turn, have designed a model for effort prediction and risk assessment in software development projects that follow the Extreme Programming (XP) methodology. The author’s approach is based on the use of Bayesian Networks (BNs), and quantitatively estimates project metrics (e.g., iterations/time to complete) without requiring data about the success of past XP projects. Fenton and Neil [12], in another research, have shown that BNs are also an effective mechanism for predicting software defects. Although relevant, these researches have only considered risks in terms of the probability of occurrence of adverse events; the severity of the impacts that such events might have on the affected projects or businesses has not been taken into account. On the other hand, Marques and Neves-Silva [13] have proposed a method for risk assessment to help in the decision making on complex assembly lines. The authors propose to compute risks – in terms of both probability and impact of possible incidents – considering information collected during the system operation. This method was designed to run in an environment where the required parameters for calculating incident’s probability and impact have well known values, for a limited set of possible events. In IT change management, however, due to the dynamics of IT environments, the amount and diversity of incidents that can happen is likely uncountable. Solutions able to cope with such a diversity is then still required. In the context of IT change management, Sauv´e et al. [5] have proposed a risk analysis method to support the scheduling of Request for Changes (RFCs). Their primary objective was to determine priorities for the implementation of potentially concurrent RFCs over a common managed IT infrastructure. The proposed method is heavily based on estimates of deployment time of RFCs and the way they can be scheduled at different moments, affecting the impact of change deployment to business objectives. Their work, however, applies to the scheduling phase of change management, and does not consider the risks associated to improper planning of RFCs, thus leaving no room for possible RFC adjustments. Aiming to deal with failures during change deployment, Machado et al. [4] proposed a solution that treats change failures in a reactive fashion, undoing the requested changes over a damaged system backwards to its previous consistent state. 1

http://www.theirm.org/

74

J.A. Wickboldt et al.

In spite of the advances, a solution that proactively observes risks to avoid future (and potentially expensive) system rollbacks is still lacking. The importance of risk assessment in IT change management lies in the fact that failures on change implementation may cause disruption of services that are relevant to business. This is underscored by the fact that some changes may look innocuous and, even indirectly, cause harm beyond their apparent complexity [2]. Oppenheimer et al. [14] have investigated several component failures in large-scale Internet services, concluding that human operator error is the leading cause of failures in these services. Automation of maintenance and operation of large-scale systems is a key factor to enhance service availability. In this context, as far as the authors of this paper are aware of, there is no automated method for estimating risks in the planning phase of change management. In the next sections, we envisage a solution for risk assessment in change planning and the way it may act as a tool to help operators designing better RFCs.

3 Automated Risk Assessment Solution In order to support risk assessment in the context of IT management, we have introduced, in a previous work [15], a new component – called Risk Analyzer – in the conceptual architecture of the C HANGE L EDGE system [3]. In this paper, we introduce and detail mechanisms, algorithms, and equations for (i) processing information collected from the IT environment and (ii) estimating risks based on metrics to quantify probability of failures and impact on affected elements. Early in this section, we review the traditional change management process, as envisaged by ITIL and materialized by the C HANGE L EDGE system; whose architecture is depicted in Figure 1. Afterwards, we present how automatic risk assessment is performed in this context. The change process starts when the Change Initiator specifies an RFC by interacting with the Change Designer component. Subsequently, the Operator sketches a preliminary Change Plan (CP), which consists of a workflow of high-level activities that describe the steps required to deliver the requested change. This workflow will be further refined by the Change Planner component. The outcome of this refinement is a CP composed of finer-grained activities that can be actually deployed over CIs [3].

Change Initiator

Change Designer

Change Authority

Operator

Change Planner

Risk Analyzer L

CI cici

R

L

Probability Estimation

Risk Classification

CI cici

Relevance Estimation

Impact Estimation

Mean Risk Calculation

R

Risk Analyzer

Configuration Mgmt. Database Change Deployer Change Management System

Log Records Generator

CICICI

Deployment System

Fig. 1. Architecture of a change management system with risk assessment support

Improving IT Change Management Processes with Automated Risk Assessment

75

In a system without risk assessment support, at this point an RFC would be ready to be approved by a Change Authority, scheduled, and deployed. However, these changes may expose the provisioned services to unnecessary or unknown risks. Therefore, the Risk Analyzer component (detailed in Figure 1 top right) automatically estimates risks in the refined CP. As input, the Risk Analyzer receives, from the Change Planner the change that will be the subject of analysis. This component also consumes the (i) execution records (list of logs L) of previous change deployments and (ii) the updated view of the IT infrastructure (list of CIs), both available from the Configuration Management Database (CMDB). By processing these inputs, the Risk Analyzer automatically generates a Risk Report (R). Analyzing this report, the Change Authority could then decide whether the risks of deploying the original RFC are acceptable or not. If not accepted, the RFC is returned to be redesigned, aiming at mitigating the reported risks. This could be done, for instance, by modifying the original workflow or the CIs affected by the chance. If the risks are considered acceptable, the CP is then scheduled and finally submitted to be deployed by the Deployment System. As mentioned, in order to estimate the probability of failures, the Risk Analyzer processes information from the execution records of RFCs. These records (following the information model proposed in a previous work [15]) are produced by the Log Records Generator during the deployment of RFCs. These records represent the execution traces of CPs obeying the original sequence of activities performed. Moreover, they include information about succeeded and failed executions; in the case of unsuccessful deployments, they include the failure classification and remediation actions taken. In the adopted information model, failures are classified into six categories: Activity Failure (AF), Resource Failure (RF), Human Failure (HF), Time Failure (TF), External Trigger (ET), and Constraint Violation (CV). In this work, however, we focus our evaluation on three of these categories: AF represents failures inherent to the activities of the CP (e.g., software installation failure); RF represents failures on the resources handled in activities (e.g., hardware damage during deployment); and HF represent the failures caused by incorrect actions taken by human operators. Some types of failures may not be easily caught by a failure detection system (especially HF). In these cases, the operator that reviews and closes RFCs should insert these records to enable future risk estimation. For impact estimation, the automated risk assessment process requires a metric that represents the importance of the CIs to business. In this work, we propose a metric called Business Relevance (BsR), which is associated to every CI that is relevant to the business continuity. BsR is expressed by a numerical value and, regardless of the scale adopted, it should enable comparisons between relevancies of different CIs. Along with the BsR, relationships and dependencies between CIs are collected from the CMDB and used for impact estimation in the risk assessment process. 3.1 Probability of Failures Estimation In order to present the behavior of the Probability Estimation module, we first introduce the definitions and metrics that support this module. One key aspect in probability estimation is the way several probabilities of failure from different RFCs compose a single probability weighted by a metric which we call Risk Affinity (RA). The goal of RA is to capture the similarities between two workflows according to a given failure type, as

76

J.A. Wickboldt et al.

shown in Equation 1. This equation uses a function θ that returns a value (ranging from zero to one) that represents the likeness of the k th pair of activities of the two workflows according to the failure type f t. In other words, the θ function considers the percentage of coincident CIs involved in pairs of activities (e.g., compares involved computers, software, and humans). However, in the case of f t been an RF, θ only returns more than zero if the activities’ actions and resources are the same (e.g., same computer). The RA metric is computed by a sum of likeness of k pairs of activities up to the size of the smaller workflow, divided by the size of the bigger one. This enables RA to capture not only local differences of activities but also to distinguish workflow sizes. min(|A|,|B|)

RA(A, B, f t) =

k=0

θk (A, B, f t)

max(|A|, |B|)

(1)

Three other functions are still required by the Probability Estimation module. The first one, called inf luences, returns a subworkflow of activities that influence a given activity a in the scope of a CP. We say that an activity b influences an activity a when b is executed either before or in parallel with a in the CP. The second function, alike enough, returns true when an activity is found to be similar to another in the context of a failure type. For example, for AFs, activities that perform the same action over the same software element are regarded as similar. The third function, possible f ailure types, returns a set of possible failure types that may happen, given and activity a (e.g., HF can only happen if a is a manual activity). The process of estimating probabilities is performed by the Algorithm 1. Intuitively, probabilities are calculated by dividing two values: (i) the sum of failure occurrences of a given activity in a set of RFCs (dividend) and (ii) the sum of the total executions of the same activity in the same set of RFCs (divisor). These two values are weighted by the RA between the analyzed CP and others extracted from the execution records. The idea is to reuse the logs from RFCs that have very similar CPs, prioritizing also similar activities that have a significant number of historical executions. In order to calculate these probabilities, the Algorithm 1 receives as input the CP of the RFC under analysis and a set of all execution records of RFCs available in the CMDB (for performance matters, this set is previously filtered matching RFCs having the same set of affected CIs as the one under analysis). Then, for each activity a of CP (Line 2), a subworkflow CP containing a and the activities that influence its execution is defined (Line 3). Following, for every possible failure type f t (Line 4), the algorithm iterates through all CPs from set R (Line 6) searching for activities that meet the alike enough criteria (Line 8). After that, RCP will be a subworkflow with the activity b and all activities that influence it in cp (Line 9). Based on CP , RCP , and a failure type the RA between both subworkflows is computed (Line 10). The result of the RA (stored in A) acts as a weight for prioritizing failure probabilities of RFCs that have similar workflows. Following, the executions and failures of b are weighted and stored in T and F respectively (Lines 11 and 12). Probability of failures for each activity and failure type are then calculated by dividing F by T (Line 13) and added to the set S (Line 14). At the end S is returned as output of the function (Line 15).

Improving IT Change Management Processes with Automated Risk Assessment

77

Algorithm 1. Probability of Failures Calculation Function Input: R: set of CPs with their execution records (logs), CP : change plan Output: set of tuples containing activity, failure probability, and failure type 1. S ← set of empty tuples (activity, failure probability, failure type) 2. for each Activity a ∈ CP 3. do CP ←inf luences(a, CP ) 4. for each FailureType f t ∈ possible f ailure types(a) 5. do T ←0; F ←0; 6. for each ChangePlan cp ∈ R 7. do for each Activity b ∈ cp 8. do if alike enough(a, b, f t) 9. then RCP ← inf luences(b, cp) 10. A ← RA(CP , RCP , f t) 11. T ← T + ( executions of b in logs of cp ∗ A) 12. F ← F + ( failures of type f t for b in logs of cp ∗ A) 13. ϕ←F ÷T 14. S ← S ∪ {a, ϕ, f t} 15. return S

3.2 Impact Estimation Another functionality of the Risk Analyzer is to estimate the impact of a change on the CIs. Initially, the Relevance Estimation module computes the Absolute Relevance (AR) of the items handled in the CP, by means of the Algorithm 2. AR is a metric that indicates the overall perception of relevance of an element to the business continuity, including its BsR and the sum of BsR of all elements that depend on it, directly or indirectly. In this algorithm, for each CI ci handled in the CP (Line 2), the value of the AR for the element ci (variable γ) is initiated with its own BsR (Line 3). Subsequently, a set D is created and populated with elements that depend, directly or indirectly, on ci (e.g., software that depend on the computer where it is hosted or services that depend on other services) (Line 4). This set is filled in recursively by iterating through dependencies defined between CIs. Following, each element that belongs to D (Line 5) will have its BsR accumulated in the variable γ (Line 6). Finally, the tuple (CI, AR) is included in the set U (Line 7), and at the end of calculation U is returned (Line 8). After AR computation, the Impact Estimation module will proceed with the normalization of these values to a metric we call Impact Factor (IF). This metric represents the portion of the infrastructure that is compromised by failure of a particular CI. The IF calculation function (Algorithm 3) receives as input the output of Algorithm 2. In order to calculate the IF of CIs, we define an element that represents the IT infrastructure, whose all CIs depend on. The AR of this element is the sum of all BsRs defined, and it is handled in all RFCs. Firstly, the algorithm instantiates the variable t with the element that represents the IT infrastructure (Line 2). Then, it invokes a pre-defined procedure that locates and extracts the CI t from the set R. (Line 3). For each tuple of the set R (Line 4), the AR from the CI contained this tuple is then divided by the AR of the whole infrastructure contained in tuple T (Line 5). Finally, a set I receives the results of these divisions (Line 6), which is returned as output of the function (Line 7).

78

J.A. Wickboldt et al.

Algorithm 2. Absolute Relevance Calculation Function Input: V : updated representation of IT infrastructure, CP : change plan Output: set of tuples containing CIs their Absolute Relevancies 1. U ← empty set of tuples (CI, AR) 2. for each ConfigurationItem ci ∈ set of handled CIs of CP 3. do γ ← BsR of ci 4. D ← set of CIs that depend on ci 5. for each ConfigurationItem d ∈ D 6. do γ ← γ + BsR of d 7. U ← U ∪ {ci, γ} 8. return U Algorithm 3. Impact Factor Calculation Function Input: R: set of tuples containing CIs and their Absolute Relevancies Output: set of tuples containing CIs and their Impact Factors 1. I ← empty set of tuples (CI, IF) 2. t ← CI that represents the whole IT infrastructure 3. T ← extract ci(t, R) 4. for each Tuple i ∈ R 5. do λ ← (AR of i) ÷ (AR of T ) 6. I ← I ∪ {ci, λ} 7. return I

3.3 Classificating and Reporting Risks The results obtained with Algorithms 1 and 3 (respectively, probability of failure and impact of change) serve as input for the classification of risks of the activities belonging to the CP under analysis, which is performed by the Risk Classification module. The main objective when automating risk assessment is to provide support for decision making on the approval of RFCs. Therefore, the results must be presented in clear and objective way. IRM [9] recommends quantifying probability and impact using the following scales: (i) high (more than 25%), medium (between 25% and 2%), and low (less than 2%), for probabilities, and (ii) high (significant), medium (moderate), and low (insignificant), for impact. The results obtained in previous steps are then mapped to these scales according to the risk classification matrix presented in Table 1. According to this matrix, each activity of the change plan may be then classified in one of nine categories. In the last step of risk assessment process, the Mean Risk (MR) of activities is calculated by the module Mean Risk Calculator. The input of this module is a set of activities classified according to the matrix from Table 1, considering each possible failure type. However, a report with several risk classifications for each activity of a CP may not be practical for a human to analyze and draw conclusions over it. For this reason, in this step, a harmonic mean of the categories of risk is calculated, resulting in a value of MR (ranging from 1 to 9 continuously) for each activity. For instance, assuming an activity of a CP that installs a software sw on a computer system cs. This activity has Activity Failure (AF) probability medium with low impact (Category 8), and Resource Failure (RF) probability low with high impact (Category 3). In this example, the MR of this activity results in a value of 4.36. The use of harmonic mean approximates the MR to the

Improving IT Change Management Processes with Automated Risk Assessment

79

Table 1. Risks Classification Matrix Probability of Failure High Impact High Impact High Impact Low Probability Medium Probability High Probability Category 3 Category 2 Category 1 Medium Impact Medium Impact Medium Impact Impact Factor Low Probability Medium Probability High Probability Category 6 Category 5 Category 4 Low Impact Low Impact Low Impact Low Probability Medium Probability High Probability Category 9 Category 8 Category 7

lowest risk category value, therefore working as a pessimistic approach, and prioritizing categories with highest risk. A Risk Report (R) is shown at the end of the automated risk assessment process, displaying activities sorted descending by MR values, having riskier activities at the top of the list.

4 Prototype Implementation In order to evaluate the technical feasibility of our solution, a prototype has been developed and incorporated into a change management system, designed by our research group, called C HANGE L EDGE. This system uses a subset of classes from the Common Information Model (CIM), proposed by the Distributed Management Task Force (DMTF) [16], to implement a representation of the managed IT infrastructure. The RFC and change plan documents are formalized using an extension of a model proposed by the Workflow Management Coalition (WfMC) [17], which was introduced and detailed in a previous work [3]. As mentioned earlier in this paper, the CIs of an IT infrastructure should have BsR values associated that represent their importance to the organization´s business. To materialize the BsR in the prototype, a metric was employed using the CIM Base Matric Definition class. This class defines a range of possible values for relevance to be applied to the managed elements, for example: High (1.00), Average (0.50), and Low (0.25). Elements that have some degree of relevance to the business continuity must have instances of Base Matric Value associated with a BsR value assigned. If no BsR value is assigned for a specific CI, the AR calculation will consider the element as irrelevant for the business (i.e., BsR zero). In order to represent dependencies between CIs, CIM defines several objects that implement relationships between items of an IT infrastructure. Some of these relationships explicitly represent dependencies, such as Service Service Dependency indicating when a service requires features from another service to work properly. Other relationships, though not necessarily representing dependencies, are considered as such by the risk analysis. This is the case of Installed Software Element, which implements a dependency of a software element to the computer system where it is hosted. In our prototype, a list of objects that represent dependencies is employed, which is iterated by the algorithm that calculates ARs of ICs.

80

J.A. Wickboldt et al.

For deployment of changes, C HANGE L EDGE makes use of a subsystem called Deployment System. It is responsible for translating the CP to be deployed into a BPEL (Business Process Execution Language) document [4]. The generated document is then submitted for execution by a Web services orchestration system called ActiveBPEL [18], which controls the execution of workflows and captures failures. Each CI of the IT infrastructure should have a management interface via Web services to be invoked by ActiveBPEL in order to implement change activities. After performing each activity, the Web service interface reports to a database: the status of implementation, failures occurred, and time elapsed in the execution of activity. For simulation purposes, each Web service implemented by the CIs produces failures pseudo-randomly, according to a uniform probability distribution, during the deployment of changes. Such failures are injected as exceptions and compel the orchestration system to interrupt the regular execution flow starting associated remediation plans. The Web services are customizable to associate different probabilities of failure for different failure types of specific CIs.

5 Experimental Evaluation In order to evaluate our solution, tests and measurements have been performed on an emulated IT environment. To measure the performance of changes, one of ITIL’s recomendation is to use a Service Disruption (SD) metric, which reflects damage to services caused by unsuccessful changes. This metric represents the time elapsed after a failure on change deployment until the system recovers the managed infrastructure. In addition, SD should consider the impact of failures over the affected services. To this end, we propose Equation 2 to calculate the SD for a given activity i of a CP. The calculation is performed by multiplying three factors: (i) Ff t,i which is the total number of failures of a type of f t found in the execution records of activity i; (ii) tf t,i representing the average time to recover the system from a failure of same type in activity i (may be obtained from the execution records of remediation activities); and (iii) IFf t,i which contains the impact factor of the CI affected by the failure of type f t handled in activity i. The sum of these values for each failure type considered in the risk estimation results in an SD metric of an activity. SDi = Ff t,i ∗ tf t,i ∗ IFf t,i (2) f t∈F T

For the case study, we assume a company that internally develops an automation software and that employs development teams divided into two areas: (i) Web interface and Web services development and (ii) persistency layer and database modeling. The system developed by these two teams has a Web interface written in Flex, Web services written in PHP running on Apache Web server, and information persisted over a MySQL database. Recently, the company has started developing a new version of this software. Therefore, both teams had their workstations updated using two RFCs, as shown in Figures 2 (a) and (b). The former sets up a Web development environment with Apache, PHP, and Flex Builder, while the latter, in addition to the Web server, required for testing purposes, also installs MySQL Server and a Workbench for SQL

Improving IT Change Management Processes with Automated Risk Assessment

81

development. We assume that both RFCs have been executed to deploy these changes over 24 workstations of two development labs (12 successful executions each RFC). Once the new version of the automation system is ready to be deployed, the IT change management team has to design a new RFC to prepare the 20 servers, on each subsidiary, to receive this new software. The RFC designed for such change, detailed in Figure 2 (c), is supposed to be deployed in all subsidiaries in two phases (being 10 subsidiaries per phase). This RFC describes that Apache, PHP, and MySQL must be installed on each subsidiary’s server. The configuration activities for the three software involved are manual, hence they must have humans associated. In this example, we define two human roles: the Senior operator, who performs MySQL and Apache configuration, and the Junior operator, who is in charge of configuring PHP. Although such RFC has never been executed (therefore it has no execution records for analysis) some of its activities have been performed a number of times in similar RFCs. Intuitively, one may realize that RFC (c) looks more like (b) than it does to (a), since RFCs (c) and (b) have 6 activities in common, while (c) and (a) have only 4. This similarity is captured by the Risk Affinity (RA) calculation (considering software, computers, and humans). For example, activity Configure PHP from RFC (c) has a RA of 0.43 comparing to Configure PHP from RFC (b) (in regards to Activity Failures), while the RA factor is 0.33 comparing to the same activity in RFCs (c) and (a). The Risk Report automatically generated for RFC (c) is illustrated in Table 2 (a). In this report, one may notice that the riskier activities are those performed by humans. Considering this report, activity Configure PHP, which is executed by the Junior operator, requires special attention. Another fact is that all MR values are between 4 and 6; this happens because all subsidiaries’ servers have medium impact. Supposing that a Change Authority has analyzed this risk report and decided to deploy the RFC as it is. In the first deployment phase, 10 of the subsidiaries’ servers have been successfully installed. By the end of this process, the total SD caused by the change deployment reaches a value of 6.68. This value is mostly influenced by activity Configure PHP,

(a) Installation/configuration of Web interface development environment

Activity

Install Apache 2 Activity

Configure Apache 2

(b) Installation/configuration of persistence development environment

Activity

Install Flex Builder 3

Activity

Install Apache 2

Activity

Configure Flex Builder 3

Activity

Configure Apache 2

Activity

Install PHP 5

Activity

Install PHP 5 Activity

Configure PHP 5

Activity

Install MySQL Server 5

Activity

Activity

Install Apache 2

Activity

Configure MySQL Server 5

Activity

Configure MySQL Workbench

Activity

Install MySQL Server 5

Activity

Configure Apache 2

Activity

Install MySQL Workbench

Configure PHP 5

(c) Installation/configuration of subsidiary servers

Activity

Configure MySQL Server 5

Activity

Install PHP 5 Activity

Configure PHP 5

Fig. 2. Change plans of installation/configuration of environments

82

J.A. Wickboldt et al.

which has the worst MR. This activity is specially harmful because it is executed in a later moment on the workflow, hence its failure causes other activities to rollback. Aiming at reducing SD for the second phase, the Operator may suggest modifications in the original CP based on the results generated by the automated risk assessment. For instance, a more experienced human could be reallocated to the riskier activity. Therefore, for the second phase, the RFC was adapted allocating the Senior operator to configure PHP and the Junior operator to configure Apache. Table 2 (b) shows the risk report of the RFC with humans reallocated. In this report, one may notice the reduction of MR in the activity Configure PHP, whereas the MR of Configure Apache increases. After the RFC is adjusted, the second phase is deployed, reaching a total SD factor of 4.11. This represents a decrease of 38.47% in the total SD when comparing phases 1 and 2, indicating that the modification of the CP based on risk assessment reports has effectively decreased the risks associated to the requested change. Table 2. Risk Reports before and after the modification of the Change Plan (a) Results before 1st phase Activity Mean Risk Configure PHP 4.86 Configure Apache 5.29 Configure MySQL 5.29 Install Apache 5.45 Install PHP 5.45 Install MySQL 6.00

(b) Results before 2nd phase Activity Mean Risk Configure Apache 4.86 Configure PHP 5.29 Configure MySQL 5.29 Install Apache 5.45 Install PHP 5.45 Install MySQL 6.00

6 Conclusion and Future Work In this work, we discussed the organization’s need for rational IT management. Since changes are imminent in such a dynamic environment, failures during this process may have direct effect on business continuity. Therefore, risks associated to changes should be investigated and mitigated. However, risk assessment has been usually left under the responsibility of humans operators, which may lead to inaccurate basis for decision making. Thus, in this paper, we proposed a solution for automating the risk assessment in IT change management, aiming to aid administrators to design better changes improving quality of change management and managed services. The results obtained, although not exhaustive, have shown that the automated risk assessment was able to combine several probabilities of failures from similar RFCs into a single probability weighted by a Risk Affinity factor. Moreover, the impact of affected CIs was considered along with probability of failures to classify activities of a CP according to a risk scale. The risk reports have shown to be useful to identify threats in a CP enabling proactive treatment of risks. Furthermore, a metric of Service Disruption was employed to compare the different CPs which revealed distinct risks reports. The mitigation of risks has caused an improvement in the SD factor, which indicates that risk reports reflect real threats to supported services.

Improving IT Change Management Processes with Automated Risk Assessment

83

In future work, we intend to investigate how to take advantage of other probability combination strategies, such as Bayesian Networks, as proposed by Hearty and Fenton. By employing such a technique an administrator could inject other factors into probabilities, such as uncertainty for RFC having very low historical information available. In addition, the case study presented in this paper has shown that human allocation to manual activities may definitely affect risks associated with changes. This leads to another question: What are the tradeoffs between different human allocations, in regard to costs, deployment time, and risks?

References 1. Office of Government Commerce (OGC): ITIL - Information Technology Infrastructure Library (2008), http://www.itil-officialsite.com/ 2. Office of Government Commerce (OGC): ITIL - Information Technology Infrastructure Library: Service Transition Version 3.0 (2007) 3. Cordeiro, W.L.C., Machado, G.S., Andreis, F.G., Santos, A.D., Both, C.B., Gaspary, L.P., Granville, L.Z., Bartolini, C., Trastour, D.: ChangeLedge: Change Design and Planning in Networked Systems based on Reuse of Knowledge and Automation. Computer Networks (2009) doi:10.1016/j.comnet.2009.07.001 4. Machado, G.S., Cordeiro, W.L.C., Daitx, F.F., Both, C.B., Gaspary, L.P., Granville, L.Z., Sahai, A., Bartolini, C., Trastour, D., Saikoski, K.: Enabling Rollback Support in IT Change Management Systems. In: 11th IEEE/IFIP Network Operations and Management Symposium (NOMS), Salvador, Brazil, pp. 347–354 (2008) 5. Sauv´e, J. P.; Santos, R. A.; Almeida, R. R.; Moura, J. A. B.: On the Risk Exposure and Priority Determination of Changes in IT Service Management. In: 18th IFIP/IEEE Distributed Systems: Operations and Management (DSOM), San Jose, USA, pp. 147–158, 2007. 6. Froot, K.A., Scharfstein, D.S., Stein, J.C.: Risk management: Coordinating corporate investment and financing policies. Journal of Finance, American Finance Association, 1629–1658 (1993) 7. Danaei, G., Hoorn, S.V., Lopez, A.D., Murray, C.J.L., Ezzati, M.: Causes of cancer in the world: comparative risk assessment of nine behavioural and environmental risk factors. In: The Lancet, vol. 366(9499), pp. 1784–1793. Elsevier, Amsterdam (2005) 8. Kl¨uppelberg, C., Kostadinova, R.: Integrated insurance risk models with exponential Levy investment Insurance Mathematics and Economics, vol. 42(2), pp. 560–577. Elsevier, Amsterdam (2008) 9. Institute of Risk Management (IRM): A Risk Management Standard, United Kingdom (2002) 10. Fewster, R., Mendes, E.: Measurement, prediction and risk analysis for Web applications. In: 7th IEEE International Software Metrics Symposium, pp. 338–348 (2001) 11. Hearty, P., Fenton, N., Marquez, D., Neil, M.: Predicting Project Velocity in XP Using a Learning Dynamic Bayesian Network Model. IEEE Transactions on Software Engineering 35(1), 124–137 (2009) 12. Fenton, N.E., Neil, M.: A critique of software defect prediction models. IEEE Transactions on Software Engineering 25(5), 675–689 (1999) 13. Marques, M., Neves-Silva, R.: Risk Assessment to Support Decision on Complex Manufacturing and Assembly Lines. In: 5th IEEE International Conference on Industrial Informatics, pp. 1209–1214 (2007)

84

J.A. Wickboldt et al.

14. Oppenheimer, D., Ganapathi, A., Patterson, D.A.: Why do Internet services fail, and what can be done about it? In: 4th USENIX Symposium on Internet Technologies and Systems (USITS), Seattle, USA (2003) 15. Wickboldt, J.A., Machado, G.S., Cordeiro, W.L.C., Lunardi, R.C., Santos, A.D., Andreis, F.G., Both, C.B., Granville, L.Z., Gaspary, L.P., Bartolini, C., Trastour, D.: A Solution to Support Risk Analysis on IT Change Management. In: 11th IFIP/IEEE International Symposium on Integrated Network Management (IM), New York, NY (to appear, 2009) 16. Distributed Management Task Force (DMTF): CIM - Common Information Model (2009), http://www.dmtf.org/standards/cim 17. Workflow Management Coalition (WfMC): Workflow Process Definition Interface - XML Process Definition Language (2009), http://www.wfmc.org/xpdl.html 18. Active Endpoints: ActiveBPEL Open Source Engine (2008), http://www.activebpel.org

A2A: An Architecture for Autonomic Management Coordination Alexander V. Konstantinou1 and Yechiam Yemini2 1

IBM T.J. Watson Research Center 19 Skyline Drive, Hawthorne, NY 10532, USA [email protected] 2 Columbia University New York, NY 10027, USA [email protected]

Abstract. A central challenge of autonomic systems is how to discover, monitor, analyze and control conﬁguration data to assure operational integrity. Current architectures for conﬁguration data management focus on federating repositories that are loosely synchronized, and do not oﬀer autonomic coordination services. We present A2A, a novel autonomic peering architecture which delivers a uniﬁed and consistent view of actual element conﬁguration for autonomic systems and managers, and provides synchronization primitives enabling policy coordination and mediation. We discuss the diﬀerent synchronization semantics and protocols used by systems and managers to access and manipulate conﬁguration data stored in a distributed Modeler. We show how dependent or conﬂicting policy actions can be automatically detected, correlated and brought to mediation. The A2A architecture has been partially implemented in a large prototype system that has been successfully demonstrated in security, network conﬁguration, and active network applications.

1

Introduction

In traditional manager-agent architectures [1,2,3,4,5] conﬁguration data is stored in a Conﬁguration Management Database (CMDB). The CMDB is populated by discovery agents which loosely synchronize its data with the actual conﬁguration state of the systems being managed. In a typical ITIL [6] process managers trigger discovery to populate the CMDB and then compare the authorized (expected) conﬁguration against the actual (discovered) conﬁguration. When unauthorized conﬁguration drifts are detected a Request for Change (RFC) is generated to update the primary local conﬁguration of the eﬀected systems. At the next discovery cycle these changes are picked up and veriﬁed so that the RFC ticket can be closed. An autonomic system is one that is self-conﬁguring, self-optimizing, selfhealing and self-protecting. The manager-agent approach to conﬁguration management is particularly ill-suited for such systems. The self-management actions of autonomic systems quickly invalidate the discovered information stored in C. Bartolini and L.P. Gaspary (Eds.): DSOM 2009, LNCS 5841, pp. 85–98, 2009. c IFIP International Federation for Information Processing 2009

86

A.V. Konstantinou and Y. Yemini

the CMDB, complicate the determination of what represents an unauthorized drift, and can immediately override any RFC actions which conﬂict with their policies. Autonomic system policies are often driven by changes in the state of other systems. Therefore a central challenge of an autonomic system is how to discover, monitor, analyze and control conﬁguration data to assure operational integrity. An autonomic system may be misled by outdated conﬁguration data in the CMDB. It may thus be unable to synchronize its conﬁguration change transactions with the underlying managed systems and the models of their state stored in the CMDB. Furthermore, the autonomic system may be unable to coordinate its conﬁguration changes with similar actions by other autonomic systems. These various potential inconsistencies between the state of the system, the CMDB model of this state, and actions by autonomic components may lead to signiﬁcant self-conﬁguration errors and loss of operational integrity. In this paper we introduce a novel peering architecture for autonomic conﬁguration management. In our A2A architecture the traditional roles of manager and agent are uniﬁed as peers accessing a distributed conﬁguration model. Managed elements maintain a local object repository (Modeler) which is accessed through a uniﬁed set of transactional local or remote interfaces. In this approach, even though management functions may continue to be distributed across local agents and remote managers, their interactions can be monitored, coordinated and mediated. Recent works [7,8,9] have considered applications of peering technologies to support discovery and distribution of management data. The primary contribution of this paper is in extending peering semantics beyond discovery and access of shared data, to support safe distributed conﬁguration access for policy veriﬁcation and conﬁguration propagation. In Section 2 we present our A2A autonomic peering architecture in relation to the current manager-agent architecture. In Section 3 we elaborate on the semantic coordination functions of the A2A Modeler. In Section 4 we link the architecture to patterns for building manageability by design, and declarative policies supporting static analysis. In Section 5 we brieﬂy discuss the NESTOR prototype implementation. Section 6 discusses related work, followed by some brief conclusions in Section 7.

2

A2A Peering Architecture

We will use a simple conﬁguration propagation example to describe the architecture and operations of the A2A peering architecture and contrast it with current manager-agent solutions. The propagation in our example will be the port and address conﬁguration of a web-server to its clients. As depicted in Figure 1 the operating system C1 hosts an HTTP server B1, requiring client A1 to conﬁgure the host and port through which they access the service. Consider ﬁrst a federated solution based on the proposed Conﬁguration Management Database Federation (CMDBf)[5] architecture shown in Figure 1. A Manager, depicted at the top left, needs to enforce the conﬁguration policy by propagating conﬁguration changes on the web-server to its respective clients. The

A2A: An Architecture for Autonomic Management Coordination

Manager (PEP) A1.outPort Å B1.inPort A1.outHost Å B1.host.name

(1) Federating (2) CMDB

A1 : Proxy outPort=80 outHost=www

87

B1: Apache inPort=80 C1 : OS name=www

MDR (3)

HTTP Proxy Apache SUSE Linux Discovery Agent Discovery Agent Discovery Agent A1 address & credentials

B1 address & credentials

WSDL(4)

SNMP

(5) Internal API A1 : Client serverPort=80 serverHost=www

Internal API B1 : Apache port=80

Java

WMI

Mgmt Agent httpd.tcp.in.port=80 os.hostname=www

C++

JMX

Mgmt Agent tcp.out.port=80 tcp.out.host=www

MIB

(6)

C1 address & credentials

Host

Internal API C1: SUSE Linux

Host

hostname=www

C

Type A Provisioning Agent

Fig. 1. Manager-Agent Architecture Example

Manager must ﬁrst discover the clients and servers whose conﬁgurations may be mismatched. This is accomplished through a federated CMDB (1). The CMDB, in turn, accesses Management Data Repositories (MDRs) to obtain the required conﬁguration data. The MDR collects conﬁguration data from the managed elements (A1, B1 and C1) through various Discovery Agents (3) which use a variety of protocols (4) (WSDL, SNMP and SSH) to discover managed elements and extract the conﬁguration data from their internal conﬁguration repositories. Once the Manager obtains the conﬁguration data, it needs to propagate the policy changes to the managed elements via respective Provisioning Agents (6) which may in turn utilize a variety of protocols and subsystems to eﬀect conﬁguration changes at the client A1. It is simple to see that this fragmented process can be highly sensitive to the ﬂow of conﬁguration data and changes, resulting in complex failures. The MDR data may be outdated, leading to erroneous Manager decisions and actions. Synchronizing the data collection activities of the Discovery Agents with the Manager decision processes and the conﬁguration changes by the Provisioning Agent is impractical. The conﬁguration changes eﬀected by the Manager may conﬂict with concurrent conﬁguration changes by other Managers. Worse, conﬁguration changes by the Manager may conﬂict with internal conﬁguration control processes of the managed elements. The conﬁguration changes by the Manager may trigger a cascade of additional conﬁguration changes by other Managers and elements. Such cascading can lead to cyclical loops of changes and non-deterministic operational behaviors, which may trigger problem management activities. For example, a self-conﬁguration action on the client to use a well-known port may be triggered if its port is changed and immediately override

A.V. Konstantinou and Y. Yemini

Data Modeling Layer

Autonomic System

Manager

Local API Events Transactions

Local API Events Transactions

Security/Views Object Repository

Peering API

Management Layer

Peering API

88

Modeler

Security/Views Object Repository Modeler

Fig. 2. A2A Architecture

the value. On the next discovery cycle the Manager would re-detect the drift and reconﬁgure, leading to a cycle. It is thus practically impossible to establish semantically consistent self-conﬁgurations through a manager-agent architecture. Our A2A autonomic peering management architecture, considered ﬁrst in [10] is depicted in Figure 2. The A2A architecture organizes autonomic peers, typically, but not necessarily embedded in managed elements, into a two-layer architecture. At the bottom layer, a distributed object Modeler, similar to the CMDB, provides a consolidated element data repository, including conﬁguration, relationship, state and performance attributes as well as their behavior events. Modeler objects are instances of classes declared in a uniﬁed management Model, such as CIM [11]. The Modeler provides a local North-South API to transactionally access and manipulate the managed data, and subscribe for events. This enables the management layer, above, to access a uniﬁed data model, interpret its behavior and activate autonomic control functions. The remote East-West API is used to federate with other Modelers to support access to conﬁguration information in remote object repositories. Existing P2P management protocols [7,8,9,12] can be used to aﬀect discovery and DHT-based sharing of distributed conﬁguration data. An A2A architecture removes the primary storage of a system’s conﬁguration from the system’s code and maintains it in the Modeler. It then oﬀers a common set of transactional semantic interfaces to the Modeler enabling peers to access and manipulate conﬁguration data and coordinate these transactions among them. This creates a peering relationship among all processes, both local and remote, wishing to transact with element conﬁguration data based on common semantic abstractions of conﬁguration objects and their manipulations. We use Figure 3 and the scenario of enforcing the simple propagation policy of Figure 1 to illustrate our A2A mechanisms. The ﬁrst key diﬀerence is that the managed elements do not store their conﬁguration internally. Instead they retrieve and update it in a transactional manner from their local Modeler. For example, the B1 Apache HTTP server system will bind its server socket to the port based on its conﬁguration in the Modeler. Should that conﬁguration change, it subscribes for an event that will trigger a rebinding of the socket. In the A2A architecture management functions can be ﬂexibly distributed among systems. The function to maintain consistent conﬁguration between the client A1 and the web-server B1 can be evaluated and maintained on the client system

A2A: An Architecture for Autonomic Management Coordination transaction { new Socket( A1.getOutPort(), A1.getOutHost()); } subscribe(A1, B1, C1, sync) { A1.outPort = A1.dependency.inPort; A1.outHost = A1.dependency.host.name; } A1 Client System

A1 : Client outPort=80 outHost=www A1 Modeler

depend

transaction { openSocket( B1.getInPort(), B1.getHost().getName()); subscribe(B1, B1.getHost()) { reopenSocket(…); } }

subscribe(C1, sync) { allHostedPortsUnique() }

B1 Apache System

C1 OS System

B1: Apache host inPort=80

89

host

B1 Modeler

C1 : Linux name=www C1 Modeler

Fig. 3. A2A Architecture Example

itself as a self-management action, or in an external manager. In either case, the veriﬁcation will be associated with a transaction to change the conﬁguration of A1 or B1 and synchronously ensure that any change is propagated to the client. The propagation policy may be programmed declaratively using an objectspreadsheet language such as OSL[13]. Once the A1 manager has computed the appropriate policy decisions it will join the update transaction and set the value of the host and port conﬁguration for A1. This update will generate a Modeler event, which the system implementation code will trap to trigger a rebinding of the client socket to the correct host and port. This semantic peering architecture for autonomic systems oﬀers several signiﬁcant advantages over the traditional manager-agent (client-server) organization. It moves conﬁguration data from local internal repositories to a uniﬁed distributed modeling layer and eliminates the need to maintain replicated repositories at systems, agents and CMDBs, and to synchronize them across diﬀerent data models and access protocols. The common transactional semantics, provided by the Modelers, enable direct shared access to conﬁguration data among all interested components, whether local or remote. This permits coordination and synchronization by distributed systems and management components of conﬁguration access and changes. Furthermore, the shared transactional semantics enable checkpointing of conﬁguration states, analysis of failures and recovery. The peering architecture permits ﬂexible scaling and changes of conﬁguration management. New elements, managers, systems components and autonomic policies can be easily joined, or removed involving modular relatively simple changes. Moreover, the peering architecture permits robust operations of autonomic management. The management system can continue to operate under dynamic changes, failures and partitioning of the network. It can eﬀect autonomic policies that gracefully reduce unavailable services and reconﬁgure resources to support self-healing of services that may be sustained through the

90

A.V. Konstantinou and Y. Yemini

failure. The uniﬁcation of the traditional roles of manager and element allows management functions to be ﬂexibly distributed at diﬀerent elements, supporting autonomic behavior. In what follows we describe in details the A2A mechanisms for semantic peering.

3

Distributed Modeler

A central function of the A2A architecture is to enable peer managers to access, share and manipulate distributed conﬁguration data. This function is primarily handled by the distributed Modeler. The Modeler provides (a) a repository of object-relationship data models and meta-models; (b) mechanisms for repository access and manipulations supporting view abstractions and security protections; (c) mechanisms to support distributed transactions by local and remote managers, and (d) publish/subscribe mechanisms to handle events. We illustrate the Modeler operations using an autonomic policy maintenance scenario depicted in Figure 4. In response to a change in the environment or its policy an autonomic manager (mgr2) determines that the outPort of B1 should be changed from 80 to 81. Mgr2 creates a write transaction (t2) to update the state of B1.inPort in the Modeler b1m. A second autonomic manager mgr1 has a synchronous event subscription on this property and will join the transaction t2. Once mgr1 has joined, the set operation of mgr2 returns, and mgr2 indicates that it is prepared to commit. In parallel, mgr1 evaluates its own policy which computes a change in A1.outPort. Mgr1 contacts Modeler a1m which also joins the transaction t2. Modeler a1m records the change to A1.outPort in the context of transaction t2. The change does not involve mgr2, and thus the propagation is complete and all parties can vote “prepared” to commit. The transaction manager will inform the participants of their joint decision to commit, and collect their acknowledgments.

Fig. 4. Conﬁguration Propagation Example

A2A: An Architecture for Autonomic Management Coordination

3.1

91

Object-Relationship Repository

The use of object-relationship models for the conﬁguration of systems[14] and networks[15] has been widely adopted in industry[16,17], and applied to autonomic systems[18]. Autonomic object models can encapsulate both conﬁguration as well as performance instrumentation data, as described in [10]. For example, a model for an IP interface can represent both conﬁguration information, such as address and netmask, as well as performance information, such as number and average size of transmitted packets. Relationships express conﬁguration, and hence operational, dependencies between systems. The network data model has been studied extensively in the context of database systems[19]. The A2A object-relationship repository can thus be implemented over proven distributed object-oriented database technologies. Management repositories model physical and software systems that can be moved, rewired, or fail. For example, the network cable connecting a server to a switch can be moved to another port. A hard drive in a RAID array can fail, reducing the overall reliability of that system. One can therefore consider managed systems as being described by a number of models. One model is that of the actual conﬁguration which represents the current state of the system. Another model represents the authorized conﬁguration, which represents the expected state of the system. Due to physical reconﬁgurations or failures, the authorized conﬁguration may not match the actual conﬁguration, and a traditional database rollback may not be feasible. In such cases, it is necessary to enter a mediation process to compute a desired state model[20,21] which represents a new valid actual state. Provisioning of this desired state model may involve automated and manual tasks for reconﬁguring existing systems and deploying new ones as reported in [22].

3.2

Distributed Transactions and Events

Autonomic systems and managers must assure semantic consistency of conﬁgurations. This requires evaluation of self-conﬁguring policies over a consistent view of conﬁguration. Managers must also coordinate conﬁguration changes used to enforce their policies. In contrast to traditional DBMS, a range of transactional semantics must be supported to access the diﬀerent conﬁguration models. The actual conﬁguration model can only be modiﬁed by local instrumentation code, and cannot be locked by managers. For example, the instrumentation of a system’s Ethernet port will update the MAC address of its peer in the Modeler. Negotiation over the desired state model can be performed atomically, in isolation, and persisted durably, but may not be isolated from concurrent changes in the actual model. For example, in the process of evaluating a security policy based on the MAC address of an Ethernet port, the connection may be physically severed. Finally, in the process of provisioning a desired state model, uncontrolled threads of change and failures may invalidate design assumptions, or partially conﬁgure the desired state.

92

A.V. Konstantinou and Y. Yemini

When policies are evaluated against the actual conﬁguration model, they must be protected against controlled as well as uncontrolled changes. Policy evaluation does not change the actual model, and thus is inherently re-entrant. An optimistic concurrency approach can therefore be taken, whereby the policy is reevaluated if a change is detected during policy evaluation. In the Ethernet port example, the optimistic lock on the conﬁguration of the peer’s address will fail, and the policy will need to be reevaluated over the new value. Any number of optimistic concurrency control algorithms can be used for this purpose[19]. Policy violations will involve reconﬁguration actions by the autonomic systems and managers. These actions must be coordinated to prevent inconsistencies and propagation loops. For example, a policy to propagate the port of a server to all clients may conﬂict with a policy to use well-known ports for client communication. In the A2A architecture, coordination is achieved in the context of a distributed transaction over the desired state model. The shared transaction becomes a Space[23] which autonomic systems can query for other systems and managers, their planned conﬁguration actions, and the policies which triggered them. Policy conﬂicts can be detected, correlated to their source, and mediated. The speciﬁc mechanisms used for mediation are beyond the scope of this paper. The mechanism used to form a shared transaction between interacting autonomic systems and managers is based on a synchronous event service. When policies are veriﬁed against the current state using an optimistic transaction, systems and managers can subscribe for changes to the actual or desired state of the objects, attributes and relationships they have accessed. For example, the manager verifying the policy over the use of well known ports will register for changes to the port attribute of client types in all objects of a speciﬁc management domain. Event subscriptions can be synchronous or asynchronous. Synchronous event notiﬁcations are performed in the context of the transaction which triggered them, and allow subscribers to join in that transaction. In our example, when a management process changes the port of the server, this will trigger a synchronous event to the manager maintaining the server to client propagation policy. Failure to notify a synchronous subscriber is treated as a vote to abort the transaction. Standard publish-subscribe architectures and protocols can be utilized[24] for message delivery. Arranging for all autonomic managers with policies operating on the same changed data to join a distributed transaction is one aspect of coordination. The second aspect is the distributed transactional protocol employed. The protocol must be resilient to failures of systems. In our example, when the manager enforcing the server to client port propagation policy receives a synchronous event, it joins the transaction. Upon evaluating its policy, it determines that the client port must be changed, and associates additional changes with the transaction. The change to the client port will notify the manager of the wellknown client port policy who will recognize a conﬂict, and identify the policy which triggered the cascading failure. At that point, the transaction can be aborted, or mediation between the two conﬂicting policies can be enforced. At

A2A: An Architecture for Autonomic Management Coordination

93

any point during this transaction, one of the managers can fail. The transaction protocol must assure consistency and recovery from such failures. The three-phase commit protocol (3PC) supports distributed coordination with a quorum-based recovery procedure when failures are detected[25,26]. Figure 5 depicts a modiﬁed state diagram for a 3PC participant. Upon a change in the actual model, or the change action of another manager over the desired state, eﬀected managers are joined to the transaction. All managers start in state R. They apply their policies over the respective Modeler repositories concurrently. Each Modeler maintains a read set and a write set for every participant. When read → write conﬂict is detected, it is necessary for the policy which read the overridden value to be reevaluated. We modify the 3PC protocol by adding a new message called restart. A restart message to the coordinator with all participants in the R or V triggers a restart message to the participants speciﬁed in the restart message. The identiﬁed participants in the V state will transition back to the R state and will re-evaluate their policies. Cyclical write propagations or otherwise inconsistent policies leading in write → write conﬂicts can result in restart loops. A transaction coordinator is responsible for detecting such loops and triggering policy mediation. vote prepared

R restart prepare abort

V

prepare ack

R

vote abort

A

V

A

C

W

C

abort

commit W Normal Transitions

Timeout Transitions

Fig. 5. Restartable 3PC Protocol

The architecture of the Modeler’s event and transaction mechanisms supports a variety of optimizations. Proﬁling of autonomic systems and policies can be used to optimize the evaluation order of interacting policies. The results of conﬂicting policy mediation can produce composite policies that will prevent future conﬂicts. Policy cycles across transactions, due to undetected propagation paths, can similarly be correlated, triggering cross-transaction policy mediation. 3.3

Security

Access to detailed system conﬁguration models is a major security concern. Intruders can use such information to quickly identify architectural and system vulnerabilities. Conﬁguration information can also be utilized in social engineering attacks. Obtaining the ability to change conﬁguration can usually be used

94

A.V. Konstantinou and Y. Yemini

to compromise or attack a system. By separating the conﬁguration repository from the system’s operational code, the A2A architecture can reduce the risk of unauthorized conﬁguration changes. Traditional database view-based security and policy-based security approaches[27] can then be enforced directly over the network model layer. The A2A architecture can be used to associate role credentials over autonomic system containment and communication paths. For example, Modeler credentials can be communicated over secure application-level communication channels.

4

Manager Layer

In the A2A management layer, autonomic systems and managers operate as peers over a shared conﬁguration model which is accessed through the Modeler described in the previous section. Autonomic systems must be designed and implemented to integrate with the Modeler. As such, all their conﬁguration attributes must be exposed through a meta-model that is instantiated and persisted within the Modeler. Autonomic element instrumentation functions for performance and physical conﬁguration attributes must be separated from the system’s autonomic functions. Beyond the instrumentation aspect, and the use of local APIs to access the Modeler, there is no diﬀerence between the built-in autonomic functions of a system and those of external managers. The management layer does not impose any additional restrictions on the design of autonomic systems and managers. The distributed Modeler layer is the foundation of two key autonomic manager technologies on which we have previously reported. JSpoon[10,13] is a language for integrating the conﬁguration and instrumentation aspects of autonomic agents and managers at design time. JSpoon provides native transactional and event primitives to support patterns of instrumentation and management access. The JSpoon runtime interfaces to the Modeler distributed transaction and event functions to support the language features. The Object Spreadsheet Language (OSL)[13] is a declarative expression language extending OCL[28] to encode conﬁguration propagation policies. By leveraging the A2A Modeler’s transactional and event interfaces, a distributed incremental OSL interpreter was built with support for static as well as dynamic propagation path conﬂict analysis and mediation.

5

NESTOR Prototype

The A2A peering architecture outlined in this paper has been partially implemented in a large research prototype called NESTOR[13]. The prototype includes a custom distributed transactional object Modeler, an object-relationship modeling deﬁnition language compiler, an incremental Object Spreadsheet Language (OSL) change rule and Object Policy Language (OPL) constraint interpreter, adapters for diﬀerent management protocols and elements, and a rich management graphical editor. The NESTOR transactional model is based on

A2A: An Architecture for Autonomic Management Coordination

95

optimistic concurrency control for instrumented conﬁguration attributes, and 2PC for desired conﬁguration state. The R3PC algorithm presented in this paper was not integrated into the platform. NESTOR was developed in two successive versions which provided practical experience with diﬀerent automation architectural designs. NESTOR has been applied to the management of security in dynamic networks[29], automating conﬁguration of network virtualization[30], instrumenting Active Network Nodes, and conﬁguring a distributed ﬁrewall based on security policies[31].

6

Related Work

Several recent works [13,12] have thus explored distributed architectures for autonomic systems. A distributed management architecture, ﬁrst proposed at [32], enables autonomic components to assure consistency of their views and actions with the actual states of managed subsystems, by directly manipulating local repositories of these subsystems. The novelty of our approach is that we focus on the safe distribution and transactional manipulation of element conﬁguration over which existing mechanisms for distributed policy enforcement and collection can be layered. A recent draft speciﬁcation for a CMDBf federation standard[5] is an attempt to standardize integration based on a federated architecture. In this architecture, conﬁguration data is aggregated in a federated database which is populated by pull or push of data from distributed Management Data Repositories (MDRs). The standard deﬁnes a graph-based query language for performing queries over the federated database that can be distributed to the MDRs. The CMDBf architecture does not challenge the basic design goals of the CMDB, and therefore does not address transactional access, synchronization, or provisioning. The A2A architecture refactors the federation and query concepts of CMDBf into a twolayer peering management architecture to support autonomic services. The challenge of data center management automation has been the subject of a number of previous studies [18,33]. A common assumption in a number of these studies is that a consistent view of the world is an input to the system. Often, the stated goal of these systems is to enforce centrally deﬁned functional and nonfunctional policy constraints. Automation is thus introduced at the management layer and assumes that the managed systems are not self-managing or have limited and well deﬁned autonomic functions. Based on these assumptions the manager can plan the changes required over the consistent world view input to provide or optimize some function, and then schedule the provisioning of these changes. The A2A architecture is an enabling technology for these studies, because it provides an essential mechanism for obtaining a consistent world view. More importantly, it will also support distributed approaches to automation using cooperating self-managed and self-healing processes. A number of recent studies have focused on the application of P2P technologies to management [7,8,9,12]. Their emphasis has been on P2P discovery and DHT-based sharing to distribute data across multiple managers for scalability. The use of P2P discovery techniques has been incorporated in our A2A

96

A.V. Konstantinou and Y. Yemini

architecture. The approach to creating uncontrolled copies of conﬁguration data, however, exacerbates the synchronization challenges we have identiﬁed as a key challenge for autonomy. Our architecture speciﬁes detailed synchronization primitives which can provide safe access to distributed peers, and enable them to join in order to negotiate over their policies. Autopilot [34] is an example of a new generation of data center management systems. These systems are characterized by a small number of vertically engineered applications that were designed to scale to hundreds of thousands of homogeneous software and hardware instances. A fault-tolerant centralized device manager receives information from data center systems and monitors which is used to update a strongly-consistent current state data model. The device manager then uses a set of manually determined policies to compute the desired data center state model. Management services and systems respectively provision and reconﬁgure based on their determined desired state. The architecture supports weak-consistency of the deployment state during provisioning. Our A2A architecture can be viewed as a ﬁrst attempt at bridging the manager-agent and vertical types of data centers, supporting both strong and weak consistency models.

7

Conclusions

The current trends towards delivery of software as a service are shifting management complexity from client systems into mission critical data centers which are rapidly evolving and enlarging. Within the data center there have been two fundamental reactions to these challenges. Enterprise data centers have focused on applying ITIL application-lifecycle technologies capturing existing best-practice workﬂows over a shared CMDB. These workﬂows tend to be static and human task oriented, admitting limited automation at predetermined points. The other reaction comes from a new generation of massive Internet applications, such as web-search, which were designed in a vertical manner, integrating autonomic policies for deployment, monitoring, failure recovery and migration. These two diverging data center architectures present diﬀerent challenges to the adoption of autonomic technologies. At the Enterprise-level, the proliferation of heterogeneous services with complex hosting and connectivity dependencies, accessed via fragmented management functions creates a high-risk environment for autonomy. Autonomic technologies will not be adopted without clear controls over what can be changed, based on what information and by whom. The current approaches towards weekly consistent CMDBs, while a big step forward, will not deliver a platform that is safe for autonomy. Our proposed A2A semantic peering architecture puts an emphasis on consistency and coordination, which are the cornerstones for eﬀective autonomic technology adoption in the Enterprise data center. The challenges posed to autonomy by massive Internet application data centers are less well understood at this point. We expect that the main challenge will be in composing the autonomic functions of diﬀerent vertical applications[35].

A2A: An Architecture for Autonomic Management Coordination

97

The widening oﬀering of services, from search to mail, and now productivity applications will challenge the vertical structure of such centers. The A2A architecture can provide a space for safe mediation of the interacting autonomic functions of these vertical semi-autonomous applications. Furthermore, it promises to unify the management architecture across types of applications, which will become increasingly important as their domains begin to overlap.

Acknowledgments Research sponsored in part by DARPA contract DABT63-96-C-0088.

References 1. ISO: OSI basic reference model - part 4: Management framework. Technical Report 7498-4, ISO (1989) 2. Case, J., Fedor, M., Schoﬀstall, M., Davin, J.: A Simple Network Management Protocol (SNMP). Technical Report RFC 1067, IETF (1988) 3. Martin-Flatin, J.: Push vs. pull in web-based network management. In: Integrated Management (May 1999) 4. Schonwalder, J., Pras, A., Martin-Flatin, J.P.: On the future of internet management technologies. IEEE Communications Magazine 41(10), 90–97 (2003) 5. CMDB Federation Workgroup: CMDB federation CMDBf. Technical report (2008) 6. Oﬃce of Government Commerce: The Oﬃcial Introduction to the ITIL Service Lifecycle Book. The Stationery Oﬃce (August 2007) 7. Zach, M., Fahy, C., Carroll, R., Lehtihet, E., Parker, D., Georgalas, N., Nielsen, J., Marin, R., Serrat, J.: Towards a framework for network management applications based on peer-to-peer paradigms the celtic project madeira. In: IEEE NOMS (April 2006) 8. Tang, C., Chang, R.N., So, E.: A distributed service management infrastructure for enterprise data centers based on peer-to-peer technology. In: IEEE Services Computing Conference (SCC) (September 2006) 9. Granville, L.Z., da Rosa, D.M., Panisson, A., Melchiors, C., Almeida, M.J.B., Tarouco, L.M.R.: Managing computer networks using peer-to-peer technologies. IEEE Communications Magazine (2005) 10. Konstantinou, A.V., Yemini, Y.: Programming systems for autonomy. In: IEEE Autonomic Computing Workshop (AMS 2003), Seattle, WA (June 2003) 11. Distributed Management Task Force (DMTF): Common Information Model (CIM) speciﬁcation. Technical Report Version 2.2, DMTF (June 1999) 12. Kamienski, C., Sadok, D., Fidalgo, J., Lima, J.: On the use of peer-to-peer architectures for management of highly dynamic environments. In: 4th IEEE Int. Conf. on Pervasive Computing and Communication (March 2006) 13. Konstantinou, A.V.: Towards Autonomic Networks. PhD thesis, Columbia University (October 2003) 14. Sloman, M.: Management for open distributed processing. DCS 1(9), 25–39 (1990) 15. Dupuy, A., Sengupta, S., Wolfson, O., Yemini, Y.: Netmate: A network management environment. IEEE Network Magazine (1991) 16. DMTF: Common Information Model (CIM). Technical report, DMTF (2006) 17. W3C: Service Modeling Language, version 1.0. Technical report (2007)

98

A.V. Konstantinou and Y. Yemini

18. Yemini, Y., Konstantinou, A., Florissi, D.: NESTOR: An architecture for Network Self-management and Organization. IEEE JSAC 18(5) (2000) 19. Elmasri, R., Navathe, S.B.: Fundamentals of Database Systems., 5th edn. AddisonWesley, Reading (2006) 20. Eilam, T., Kalantar, M., Konstantinou, A., Paciﬁci, G., Pershing, J., Agrawal, A.: Managing the conﬁguration complexity of distributed applications in internet data centers. IEEE Communication Magazine 44(3), 166–177 (2006) 21. Arnold, W., Eilam, T., Kalantar, M., Konstantinou, A.V., Totok, A.A.: Pattern based SOA deployment. In: Kr¨ amer, B.J., Lin, K.-J., Narasimhan, P. (eds.) ICSOC 2007. LNCS, vol. 4749, pp. 1–12. Springer, Heidelberg (2007) 22. El Maghraoui, K., Meghranjani, A., Eilam, T., Kalantar, M., Konstantinou, A.: Model driven provisioning: Bridging the gap between declarative object models and procedural provisioning tools. In: van Steen, M., Henning, M. (eds.) Middleware 2006. LNCS, vol. 4290, pp. 404–423. Springer, Heidelberg (2006) 23. Gelernter, D.: Generative communication in linda. ACM Trans. Program. Lang. Syst. 7(1), 80–112 (1985) 24. Eugster, P.T., Felber, P.A., Guerraoui, R., Kermarrec, A.M.: The many faces of publish/subscribe. ACM Comput. Surv. 35(2), 114–131 (2003) 25. Skeen, D.: A quorum-based commit protocol. In: 6th Berkeley Workshop on Distributed Data Management and Computer Networks (February 1982) 26. Keidar, I., Dolev, D.: Increasing the resilience of atomic commit, at no additional cost. In: PODS 1995: Proceedings of the fourteenth ACM SIGACT-SIGMODSIGART symposium on Principles of database systems, pp. 245–254. ACM, New York (1995) 27. Sloman, M., Lupu, E.: Security and management policy speciﬁcation. IEEE Network (2002) 28. OMG: Object Constraint Language speciﬁcation (OCL). Technical Report ad/9708-08 (version 1.1), Object Management Group (OMG) (September 1, 1997) 29. Konstantinou, A.V., Yemini, Y., Bhatt, S., Rajagopalan, S.: Managing security in dynamic networks. In: USENIX Lisa (1999) 30. Su, G., Yemini, Y.: Virtual Active Networks: towards multi-edged network computing. Computer Networks 36(2/3), 153–168 (2001) 31. Burns, J., Gurung, P., Martin, D., Rajagopalan, S., Rao, P., Rosenbluth, D., Surendran, A.: Management of network security policy by self-securing networks. In: DISCEX II, Anaheim, California (2001) 32. Goldszmidt, G., Yemini, Y.: Distributed management by delegation. In: The 15th Int. Conference on Distributed Computing Systems, Vancouver, BC. IEEE, Los Alamitos (1995) 33. Eilam, T., Kalantar, M., Konstantinou, A., Paciﬁci, G.: Reducing the complexity of application deployment in large data centers (2005) 34. Isard, M.: Autopilot: Automatic data center management. Operating Systems Review 41(2), 60–67 (2007) 35. Konstantinou, A., Eilam, T., Kalantar, M., Totok, A.A., Arnold, W., Snible, E.: An architecture for virtual solution composition and deployment in infrastructure clouds. In: 3rd Int. Workshop on Virtualization Technologies in Distributed Computing (VTDC) (June 2009)

Consistency of States of Management Data in P2P-Based Autonomic Network Management J´eferson Campos Nobre and Lisandro Zambenedetti Granville Institute of Informatics – Federal University of Rio Grande do Sul Av. Bento Gonc¸alves, 9500 - Porto Alegre, RS, Brazil {jcnobre,granville}@inf.ufrgs.br

Abstract. Autonomic network management is a vision that brings autonomic computing principles to network management. Besides, it is necessary some level of decentralization to enable broad autonomic capabilities. An interesting alternative of infrastructure for this union is the utilization of peer-to-peer (P2P) overlays. However, the consistency of states of management data among peers is an important challenge. Traditional mechanisms to maintain consistency of these states are supported by some centralization which wastes some desirable properties of P2P facilities. In contrast to these mechanisms, we propose a distributed, scalable and robust mechanism to maintain the consistency of states of management data. In this paper we introduce multi-agent truth maintenance features in P2P-based autonomic network management as a mechanism to bring consistency maintenance of these states. We developed a model of a P2P-based autonomic network management system on Peersim to perform simulation experiments. Besides, the utilization of P2P-based autonomic network management systems in access networks is presented as a case study.

1 Introduction The increasing complexity of computer networks requires sophisticated solutions to manage the underlying communication infrastructure and help network human administrators in their daily tasks [1]. The application of autonomic computing (AC) principles in network management, normally refereed as Autonomic Network Management (ANM), has been proposed as a way to address some demands faced by traditional network management, such as controlling highly dynamic environments like ad-hoc and peer-to-peer (P2P) networks [2]. ANM systems increase the efficiency of network human administrators by decreasing the number of manual interventions. This efficiency improvement is done by ANM systems through automation and/or optimization of some operational details of management tasks, such as fault handling and performance management. ANM systems can be deployed using different models of management distribution, from totally centralized models up to highly decentralized ones. Today, there are no clear evidences that could link the quality of autonomic actions of an ANM system with the distribution model adopted. Some authors, however, claim that some level of decentralization is required to achieve a more adequate ANM [3]. In this scenario, the typical design of decentralized ANM systems is based on a set of Autonomic Management Elements (AME) that execute management tasks and interact with one another to C. Bartolini and L.P. Gaspary (Eds.): DSOM 2009, LNCS 5841, pp. 99–110, 2009. c IFIP International Federation for Information Processing 2009

100

J.C. Nobre and L.Z. Granville

form an Autonomic Management Domain (AMD). Multiples AMDs could be integrated to form an Autonomic Management Environment. Different technologies can be employed to provide an infrastructure for decentralized ANM systems. An interesting possibility is using peer-to-peer (P2P) overlays, which incorporate characteristics of P2P networks into ANM systems, such as the support for collaborative management, robustness in connecting management entities, and load balancing of management tasks [4]. There are some initiatives investigating P2P-based ANM [5] [6] and, in these initiatives, peers have some properties found in AMEs. Besides, peer with similar properties (e.g., managing the same devices) can be organized into groups. These peer groups have some properties found in AMDs. A managed element (e.g., a network router) can be controlled by one or multiple peers in P2P-based ANM system, for example, for the purpose of management robustness. In this case, the status of the managed element becomes a distributed, replicated information among the controlling peers. Inconsistencies in this information could lead to an improper operation of the ANM system. For example, the status of a router’s link (e.g., ethernet interface) can be advertised differently by the controlling peers to external requesters (e.g., management station). This inconsistency can be caused by faults in network (e.g., losses in messages exchanged among peers) and computational resources (e.g., incorrect operation of peers). Besides, inconsistency in management data can occur even in the regular operation of P2P-based ANM system, due to lack of information synchronization among peers. This information synchronization is a challenge in unrealible asynchrounous distributed systems, such as an unstructured P2P overlay. Mechanisms to support consistency of management information in current P2Pbased ANM systems are, in general, still supported by some centralization through, for instance, external repositories [7] or super peers [5] [6], missing opportunities of unstructured P2P facilities. This centralization complicates the achievement of good scalability and robustness features, thus, in spite of being P2P-based ANM systems, these examples show undesirable characteristics of client-server systems. In this paper, we introduce Multi-Agent Truth Maintenance features [8] in the P2Pbased ANM to improve the consistency of states of management data. The introduction of these features maintains desirable scalability and robustness characteristics of P2Pbased ANM. It is also proposed a communication strategy for message exchange among peers to support this truth maintenance. This strategy uses biology-inspired processes (e.g., replication), which have well know scalability and robustness features [9]. The remainder of this paper is organized as follows. Section 2 discusses related works. Section 3 describes our proposal and its implicit concepts. Section 4 shows a case study. Section 5 shows evaluations of our proposal. Conclusions and future work are finally provided in Section 6.

2 Background In this section we first discuss about the current state-of-the-art on P2P-based autonomic network management. Afterwards, the main concepts behind truth maintenance systems are presented. Finally, some important services related to the maintenance of consistency of shared information in distributed applications are discussed.

Consistency of States of Management Data in P2P-Based ANM

101

2.1 P2P-Based Autonomic Network Management The utilization of P2P overlays is identified as an emerging approach for Autonomic Network Management (ANM). This can be specially observed in the increasing number of research work towards this utilization [10]. PBMAN [5] merge traditional PBNM with P2P overlays to autonomically manage Ambient Networks (AN). PBMAN enables scalable mechanisms for network composition inside the AN, as well as policies distribution and retrieval. Through this approach it is possible to establish policies to manage devices or services. PBMAN is structured using super peers, in a hierarchical architecture. These super peers are responsible for consistency of states of management data, among other functions, in each hierarchical level. The Madeira platform [6] is an approach to ANM that uses the concept of Adaptive Management Components (AMC), which are containers that run on managed elements. AMCs can communicate with other AMCs running on other managed elements through P2P communication services. AMCs form management clusters with super peers acting as cluster heads. These super peers are responsible for consistency of states of management data, among other functions. ManP2P [11] is a P2P-based network management system that is evolving to an autonomic conception through the implementation of autonomic modules in peers [7]. ManP2P is partially inspired by the Management by Delegation (MbD) model and based on a service-oriented approach. There is no internal mechanism for consistency of state of management data, thus, authors propose the utilization of external repositories. Despite many improvements brought by the utilization of P2P-based ANM systems, there are still issues to be addressed. The consistency of state of management data among the peers is usually addressed with some centralization, using super peers [5] [6] or external repositories [7], which misses opportunities of P2P overlays (e.g, robustness). 2.2 Truth Maintenance Systems Truth-Maintenance Systems (TMS) were proposed to keep the integrity of Knowledge Bases (KB). The origin of these systems was proposed in the 1970s, for resolutions in mono-agent systems [12]. A TMS keeps track of logical structure of the set of beliefs of agents. A belief is a member of the current set of beliefs if it has valid reasons. TMSs provide considerable power using few computational resources [13]. Thus, although not being well known outside artificial intelligence community, TMSs are used in different contexts, such as policy systems [13] and network management systems [14]. TMSs have been extended for Multi-Agent System (MAS) versions, Distributed Truth-Maintenance Systems (DTMS) [8]. In a MAS, agents must be able to maintain the integrity of their KBs, despite message exchange with other agents. This maintenace of integrity can be done by a multi-agent TMS. In an analogous way, during the operation of P2P-based ANM system, peers must be able to maintain the integrity of states of management data, despite message exchange with other peers. This similarity indicates the use of multi-agent TMS in P2P-based ANM as an interesting possibility [14].

102

J.C. Nobre and L.Z. Granville

2.3 Services for Consistency of Shared Information in Distributed Systems Services for consistency of shared information can be used as a basic building block for distributed applications. A P2P-based ANM system, as a distributed application, can appropriate some good characteristics found in these systems. ZooKeeper [15] is a coordination service for distributed applications. It exposes a simple API that distributed applications can be built upon to implement higher level services for synchronization, data diffusion, and publish-subscribe schemes. ZooKeeper uses distributed server databases for read operations, however, write operations use a “leader” server (i.e., centralized database) to assure the consistency of the database. Astrolabe [16] is a distributed information management service. It works locating and collecting the status of a set of servers and reporting summaries of this information. Astrolabe is implemented using a P2P overlay, where every peer run an Astrolabe agent (i.e., in a MAS fashion). However, Astrolabe was developed primarily using simple data models. Besides, its operation is aimed at read-oriented applications. Scalable Distributed Information Management System (SDIMS) [17] is a service to aggregate information about large-scale network systems. The service is built using ideas from Astrolabe [16] and Distributed Hash Tables (DHT). However, as in most DHT approaches, consistency and replication issues are a known challenge. The presented efforts show interesting characteristics for consistency of shared information in distributed systems. However, these efforts have vulnerabilities which make them not appropriate for P2P-based ANM systems, such as centralization [15], simple data models [16], and replication issues [17]. The consistency of states of management data is still an issue to be adressed in P2Pbased ANM systems. It is necessary to maintain this consistency keeping scalability and robustness features of P2P overlays. The maintenance of integrity of KBs among agents brought by multi-agent TMS seems to be a valid direction to introduce this consistency maintenance [14]. Besides, it is interesting to bring characteristics of services for consistency of shared information.

3 Proposal In P2P-based Autonomic Network Management (ANM) systems, peers must share management data. In this work, management datum is defined as a management information described in a defined form (i.e., using a specific language). Besides, in these systems, management data must allow their use in automation and/or optimization procedures. It is also expected that sources of management data (e.g., highly dynamic environments) impose challenges to the ANM system. Despite these requirements, it is necessary to avoid potencial inconsistencies in state of management data among peers. Our proposal is aimed at meeting requirements of consistency of states of management data in a P2P-based ANM system. The proposed mechanism introduces multiagent truth maintenance [8] features through a consistency maintenance module that runs in each peer. As far as we are aware of, the only study that incorporates multiagent truth maintenance features in a network management system was carried out by Nobre and Granville [14], and only some results were published so far.

Consistency of States of Management Data in P2P-Based ANM

103

Multi-agent truth maintenance is a multi-agent extension to Truth-Maintenance Systems (TMS). The goal of TMS utilization is to keep the integrity of knowledge bases. In this work, a knowledge base is a group of integrated management data. In a multi-agent TMS, there are multiple agents and each one has its own TMS. TMSs keep integrity performing belief revision and exchange in a set of beliefs. A belief is a member of the current set of beliefs if it has valid reasons. The exchange of beliefs about management data is done asynchronously and we do not consider the message exchange to be reliable. Unfortunately, it is well known that the utilization of asynchronous unreliable distributed systems imposes challenges to achieve consistency in shared data. Thus, the consistency model used is non-deterministic, in other words, it uses a “weak” notion of consistency. This model is adopted for scalability, robustness, and update dissemination issues. Given a belief X that depends on some other belief Y, when an update is made to Y, it is eventually reflected in X. Some authors call this notion as “eventual consistency” [16]. 3.1 Justifications for Management Data Multi-agent TMS is a kind of justification-based TMS. In a justification-based TMS, a datum is believed when it has valid justifications (i.e., valid reasons). This belief can be shared by different peers (which play the agent role) in a P2P-based ANM system (which aggregates some MAS characteristics). Thus, justifications improve the alignment of peers with system-wide objectives (i.e., objectives of the P2P-based ANM system). The datum and its list of possible justifications must be provided by network human operators or expert systems for the P2P-based ANM system. The associated states of a datum are “in” (believed) or “out” (disbelieved), according to its justifications. These justifications can be generated by processes inside the peer or received through the P2P overlay communication services. Thus, the “in” state can assume two additional states: “internal”, where the datum has only valid internal justifications, and “external”, where the datum has some valid external justification (provided by other peer). A datum is labeled “out” when it lacks, at least, one of its associated justifications [8]. For instance, the activation (belief) of a QoS policy (datum) can be justified by a network human administrator command (justification) and an asynchronous signal from a managed device (justification). The code below shows a possible internal representation of this datum and its justifications. In the example, each datum or justification (“just” in the code) is represented with 2 fields, “name” and “desc” (description). The last line of the code defines that if the justifications “adm cmd” and “async sig” are present, the datum “qos pol” is believed. datum: just: just: tms:

name qos_pol desc QoS policy name adm_cmd desc network human administrator command name async_sig desc asynchronous signal from a managed device qos_pol (adm_cmd async_sig)

104

J.C. Nobre and L.Z. Granville

3.2 Architecture of Peers Peers are commonly viewed as composed by one or more management service modules. Management service modules perform regular management tasks (e.g., collecting statistics) in each peer, and, in these tasks, modules produce management data, building management knowledge bases. We introduce the consistency maintenance module to registrate the set of belief about management data in each peer. This module works associating management data and their respective justifications. When there is a belief change (i.e., justitication change), the consistency maintenance module uses the P2P overlay communication services to spread the change. Figure 1 shows the relation between the consistency maintenance module, management service modules, and P2P overlay communication services. The management service modules should inform consistency maintenance module about their internal beliefs of management data. The management service modules are also responsible for querying and requiring services from the consistency maintenance module, possibly updating their internal beliefs. These beliefs are exchanged through justification using a simple interface that management service modules and consistency maintenance module must use. When the presence of a justification is modified, the consistency maintenance module performs the following steps: unlabels management datum, includes (or removes) the presence of justification and labels datum again according to new restrictions. The P2P overlay communication services are used to spread changes, which can change beliefs of other peers. It is important to stress that there is only one consistency maintenance module inside a peer, thus, it is not specific of a management service module. Therefore, every management service module in a peer interacts with the same consistency maintenance module. This fact can be explored for the integration of different management services.

Fig. 1. Peer architecture

Consistency of States of Management Data in P2P-Based ANM

105

For instance, a policy processing module, a fault handling module, and a configuration management module (possibly using different languages for representing management data) could be integrated by the consistency maintenance module through justifications. 3.3 Communication within a Peer Group The consistency maintenance module handles the message exchange through P2P overlay communication services. In this process, requests are adapted in messages to be spread among peers and vice-versa. The P2P overlay is modelled as an unstructured overlay networks, thus there is no relation between the information stored at a peer and its position in the overlay topology. We use the premise that there is support for group organization (i.e., peer groups) through management services modules. Thus, peers that have a specific management service module are organized into a group (without human intervention) and peers can participate of several groups accordingly to modules that they have. The methods used for message exchange inside the peer group are modeled using concepts from biology-inspired distributed computing models [9]. Among these models, proliferation-based ones are an interesting choice for communication requirements of our proposal. All peers in the peer group run exactly the same communication algorithm, which can be initiated from any peer in the peer group. We have chosen replication as the initial proliferation mechanism in the peer group. This mechanism can support a number of different strategies [9]. In our proposal, peers spread messages to replicate changes in justifications among the participating entities (i.e., peers of a specific peer group). This unbridled replication is restricted to peer group, fulfilling the criterion of robustness and controlling the number of messages within the P2P overlay.

4 Case Study The case study presented is an illustration of the collaborative fault management of links in access networks through failure notification sent by devices and human knowledge about these notifications. The integration of these information (failure notification in addition to human knowledge) produces a management datum, which can assume different states. This datum can be used against a Service-Level Agreements (SLAs) to support or clarify service level claims. Among access network technologies in metropolitan networks, Ethernet is one of most interesting and promising choice, thus, we choose it to build our case study. In this context, an access network link is an Ethernet Virtual Connection (EVC) [18]. Fault management in this link is done through Alarm Indication Signal (AIS) messages [18]. These messages are triggered when a failure between two nodes occurs. Thus, AIS messages provide asynchronous notification to other elements in the network that there is a fault in the Ethernet network. The efforts to manage layer 2 Ethernet service must consider an overlayed IP infrastructure [19]. A management service module collects AIS messages and another module collects information from human administrators. The consistency maintenance module is

106

J.C. Nobre and L.Z. Granville

responsible to integrate the information from both management service modules and maintain the consistency of the state of management datum in the peer group. The link failure detected (datum) is justified by human administrator commands from both domains, network operator detection and network consumer detection, and a device notification received (AIS message). The code below shows the representation of this datum and its justifications. These justifications are provided by management service modules and kept inside the peer group that offers this management service. datum: just: just: just: tms:

name link_fail_det desc link failure detection name net_opt_det desc network operator detection name net_con_det desc network consumer detection name ais_msg desc ais message link_fail_det (net_opt_det net_con_det ais_msg)

Initially, the datum has the “out” state, since the justifications are not present. This state can be changed after the verification of new beliefs (i.e., justification changes) by consistency maintenance module. These beliefs are informed by management service modules or received as a message in the peer group. If every justification of the datum is present, the state changes to “in”. This “in” state assumes two options according the sources of justifications: “internal” (every justification was generated internally) and “external” (some valid external justification). For instance, human administrator commands (network operator detection and network consumer detection) can be generated internally and the presence of an AIS message can be received as a justification change message. In this example, the “in” state assumes the option “external”. The code below shows the answer from the consistency maintenance module in this situation. tms:

link_fail_det:external (net_opt_det:mod net_con_det:mod ais_msg:msg)

This management operation is traditionally performed through separate centralized systems, a network management system (collecting notifications from devices), and a trouble ticket system (collecting information from human administrators). The traditional procedure brings concerns in scalability and robustness and imposes difficulties in the integration of the information. Besides, justifications can be used to provide explanations to the user [13], improving the understanding of management data.

5 Evaluation The evaluation of our proposal can be performed in different ways. To enable a fully controlled environment for the evaluation, we chose to develop some simulation experiments. In these experiments, we present simulation results that support our scalability and robustness claims. Scalability and robustness are some of the most important motivations for using decentralization in the infrastructure of different systems [20] [9], such as P2P facilities.

Consistency of States of Management Data in P2P-Based ANM

107

As previously stated, we expect that the introduction of multi-agent TMS features keeps decentralization properties of a P2P-based ANM system, maintaining each peer as an independent and self-sustainable entity. As many systems have demonstrated, a system that does not share resources can scale almost infinitely simply by adding constitutive elements (e.g, peers in a P2P-based ANM system). Besides, maintaining the independence of each peer, single points of failure are eliminated. The simulation experiments were implemented in Java using PeerSim [21], an open source event-based simulator of P2P systems. The preliminary system version used has the ability to simulate failures in peers and message exchange, and the overlay is built randomly. The experiments use a simple model of transport layer that can emulate some characteristics, such as loss and delay probabilities. All peers in peer groups run exactly the same algorithm. In the experiments, we varied the number of peers of the peer group from 4 to 14 (we do not expect large peer groups in P2P-based ANM systems). Besides, these peer group sizes seem reasonable for the case study provided in Section 4. In addition, a peer is chosen randomly as the primary source of changes to not affect measurements and message delay is controlled. Each experiment was conducted at least 10 times. In the experiments, the variance observed was low. In the first experiment, it is measured the number of messages exchanged to spread justification changes in the peer group. This number must be considered as an important cost of the peer group operation, thus, it is important for scalability analysis. Besides, we consider the number of transmitted messages as indicative of network load. In this experiment there were no faults in peers or in message exchange. We show the results in Figure 2. 30 1 justification 2 justifications 25

# messages

20

15

10

5

0 4

6

8

10

12

# peers

Fig. 2. Message exchange due to justification changes

14

108

J.C. Nobre and L.Z. Granville

Our proposal shows acceptable scalability characteristics on number of exchanged messages, since this operation (message exchange) is restricted to each peer group. The experiment shows that our system behaves like we expected, without stability and convergence problems. Network load grows linearly with the number of participating peers, thus we can infer the behavior trend of peer groups with larger number of participating peers. Of course, an efficient operation of large peer groups needs modifications in communication strategies, such as the utilization of gossip-based protocols. In the second experiment, we determined the influence of message loss on the dissemination of a justification change. In this experiment, we varied the message loss probability with following values: 25%, 50%, and 75% (respectively, 0.25, 0.5, and 0.75 as indicated in Figure 3). Using our case study, we would probably observe such message loss (specially 75%) due to faulty or overloaded network equipment (e.g., ethernet interfaces) and/or network links (e.g., ethernet lines). Since our case study is aimed at fault management (considering an overlayed IP infrastructure), our system must behave acceptably even in bad network conditions. In Figure 3, we show the average percentage of coherent (and correct) peers after message exchange to cease. The experiment shows the influence of message loss in the replication process. As can be seen from the results in Figure 3, high loss probabilities do lead to less consistency in peer group, but, even with a few participating peers, the percentage of coherent peers is substantial. Besides, more participating peers in peer group decrease the influence of loss probability. The results show some fault-tolerance features, since the peer group operation is not highly sensitive to peer crashes and message losses. But an increase in number of

110

100

90

% coherent peers

80

70

60

50

40

30

loss = 0.25 loss = 0.50 loss = 0.75

20 4

6

8

10

12

# peers

Fig. 3. Coherent peers after a justification change

14

Consistency of States of Management Data in P2P-Based ANM

109

peers also leads to an increase in the number of exchanged messages, so the robustness advantages come at some cost. However, messages are exchanged only within the peer group, and a high number of peers in a peer group is not expected. Thus, the number of exchanged messages does not impose a issue in scalability.

6 Conclusions and Future Work The support of new demands faced by traditional network management is a key research issue in network management area. One of these demands is the support for collaborative management tasks over access network links. In order to enable efficiently these tasks, it is necessary some decentralization and automation features. These features can be supported by P2P-based autonomic network management (ANM) systems. However, the consistency of states of management data imposes challenges for these systems. In this paper we have introduced multi-agent TMS features to improve the maintenance of consistency of states of management data in P2P-based ANM systems. Our proposal aims at the integration of data used by the entities that form these systems (i.e., peers), through the utilization of justifications. We have also presented evaluations of this proposal through simulation experiments. In addition, we have described a case study of fault management in access networks to show the possibilities of our proposal. Although the proposal shows good results in evaluations performed until the present moment, it is necessary to evaluate more complicated cases, in number of peers and peer groups, and in the participation of a peer in different peer groups. We are also looking at additional settings that could lead to important effects, such as network partitions. Thus, we are currently pursuing new experiments with PeerSim.

References 1. Kind, A., Dimitropoulos, X., Denazis, S., Claise, B.: Advanced network monitoring brings life to the awareness plane. IEEE Communications Magazine 46(10), 140–146 (2008) 2. Pras, A., Schoenwaelder, J., Burgess, M., Festor, O., Perez, G.M., Stadler, R., Stiller, B.: Key research challenges in network management. IEEE communications magazine 45, 104–110 (2007) 3. Jennings, B., Van Der Meer, S., Balasubramaniam, S., Botvich, D., Foghlu, M.O., Donnelly, W., Strassner, J.: Towards autonomic management of communications networks. IEEE Communications Magazine 45(10), 112–121 (2007) 4. Granville, L.Z., da Rosa, D.M., Panisson, A., Melchiors, C., Almeida, M.J.B., Tarouco, L.M.R.: Managing computer networks using peer-to-peer technologies. IEEE Communications Magazine 43(10), 62–68 (2005) 5. Kamienski, C., Fidalgo, J., Sadok, D., Lima, J., Pereira, L., Ohlman, B.: PBMAN: A Policybased Management Framework for Ambient Networks. In: IEEE Workshop on Policies for Distributed Systems and Networks, POLICY 2006, pp. 79–83 (2006) 6. Fallon, L., Parker, D., Zach, M., Leitner, M., Collins, S.: Self-forming Network Management Topologies in the Madeira Management System. In: Bandara, A.K., Burgess, M. (eds.) AIMS 2007. LNCS, vol. 4543, pp. 61–72. Springer, Heidelberg (2007) 7. Marquezan, C.C., dos Santos, C.R.P., Nobre, J.C., Almeida, M.J.B., Tarouco, L.M.R., Granville, L.Z.: Self-managed services over a p2p-based network management overlay. In: Proceedings. 2nd Latin American Autonomic Computing Symposium, LAACS 2007 (2007)

110

J.C. Nobre and L.Z. Granville

8. Huhns, M.N., Bridgeland, D.M.: Multiagent truth maintenance. IEEE Transactions on Systems, Man and Cybernetics 21(6), 1437–1445 (1991) 9. Babaoglu, O., Canright, G., Deutsch, A., Di Caro, G.A., Ducatelle, F., Gambardella, L.M., Ganguly, N., Jelasity, M., Montemanni, R., Montresor, A., et al.: Design patterns from biology for distributed computing. ACM Transactions on Autonomous and Adaptive Systems (TAAS) 1(1), 26–66 (2006) 10. EMANICS: Activities - Work Package 9 - Autonomic Management (2009), http://emanics.org/content/view/63/109/ 11. Panisson, A., da Rosa, D.M., Melchiors, C., Granville, L.Z., Maria, L.: Designing the Architecture of P2P-Based Network Management Systems. In: ISCC 2006: Proceedings of the 11th IEEE Symposium on Computers and Communications, pp. 69–75. IEEE Computer Society, Los Alamitos (2006) 12. Doyle, J.: A truth maintenance system. Computation & intelligence: collected readings, 529– 554 (1979) 13. Kagal, L., Hanson, C., Weitzner, D.: Using Dependency Tracking to Provide Explanations for Policy Management. In: IEEE Workshop on Policies for Distributed Systems and Networks, POLICY 2008, pp. 54–61 (2008) 14. Nobre, J.C., Granville, L.Z.: Towards consistency of policy states in decentralized autonomic network management. In: IEEE Workshop on Policies for Distributed Systems and Networks, POLICY 2009 (to appear, 2009) 15. Hunt, P.: ZooKeeper: A Distributed Coordination Service for Distributed Applications (2008), http://wiki.apache.org/hadoop/ZooKeeper 16. Van Renesse, R., Birman, K., Vogels, W.: Astrolabe: A robust and scalable technology for distributed system monitoring, management, and data mining. ACM Transactions on Computer Systems (TOCS) 21(2), 164–206 (2003) 17. Yalagandula, P., Dahlin, M.: A scalable distributed information management system. ACM SIGCOMM Computer Communication Review 34(4), 379–390 (2004) 18. McFarland, M., Salam, S., Checker, R.: Ethernet oam: key enabler for carrier class metro ethernet services. IEEE Communications Magazine 43(11), 152–157 (2005) 19. Ryoo, J., Song, J., Park, J., Joo, B.S.: Oam and its performance monitoring mechanisms for carrier ethernet transport networks. IEEE Communications Magazine 46(3), 97–103 (2008) 20. Mccann, J.A., Huebscher, M.C.: Evaluation Issues in Autonomic Computing. In: Jin, H., Pan, Y., Xiao, N., Sun, J. (eds.) GCC 2004. LNCS, vol. 3252, pp. 597–608. Springer, Heidelberg (2004) 21. Jelasity, M., Montresor, A., Jesi, G., Voulgaris, S.: The Peersim Simulator (2008). http://peersim.sf.net

Self-optimizing Hybrid Routing in Publish/Subscribe Systems Arnd Schr¨ oter, Daniel Graﬀ, Gero M¨ uhl, Jan Richling, and Helge Parzyjegla Communication and Operating Systems Group (KBS), Berlin University of Technology, Germany {aschroet,dgraff,gmuehl,richling,parzy}@cs.tu-berlin.de

Abstract. Rendering networks and distributed systems self-managing and self-optimizing has become a major research focus. This task is especially important for systems, such as publish/subscribe systems, that are used in dynamic environments. In these settings, a static conﬁguration usually leads to a largely suboptimal system performance, while manual optimization is either too expensive or not possible at all. Distributed publish/subscribe system are usually realized by a broker overlay network providing the functionality of a decentralized notiﬁcation service. In this paper, we present an approach that self-optimizes the routing conﬁguration of a publish/subscribe broker overlay network to minimize the operational costs of the system without any manual intervention. It is based on a new class of routing algorithms, which allows a link-bylink adaptation of the routing conﬁguration at runtime. Based on a local decision criterion, a decentralized optimization algorithm is introduced that generates only marginal extra traﬃc. Keywords: Publish/Subscribe Middleware, Self-Optimization.

1

Introduction

Today, many administrative tasks in networks and distributed systems are still executed manually by human operators. However, due to the increasing complexity and dynamics of these systems, this cost-extensive approach comes to a limit. Because of this, a major research focus of the recent years was to render these systems self-managing such that they can, for example, self-optimize their conﬁguration to maximize the system’s performance without human intervention. Publish/subscribe systems provide an asynchronous, anonymous, and datacentric many-to-many communication model and decouple the communication partners in time, space, and ﬂow [5]. Leveraging this ﬂexibility, publish/subscribe infrastructures are increasingly often used to build novel distributed systems and applications ranging from information and content dissemination over distributed event processing to system integration and monitoring. Publish/subscribe systems are especially interesting in the context of self-managing systems for two main reasons: First, they are an ideal basis for realizing autonomic and C. Bartolini and L.P. Gaspary (Eds.): DSOM 2009, LNCS 5841, pp. 111–122, 2009. c IFIP International Federation for Information Processing 2009

112

A. Schr¨ oter et al.

self-managing distributed systems and, second, because of their complexity they should be made self-managing, too. Components within a publish/subscribe system communicate by producing and consuming notiﬁcations. While producers aka publishers publish notiﬁcations, consumers aka subscribers subscribe for notiﬁcations they are interested in (e.g., using content-based ﬁltering). The publish/subscribe system acts as notification service, which connects the components and delivers published notiﬁcations to all consumers with matching subscriptions. Publish/subscribe middleware systems (e.g., Hermes [9] and Rebeca [6]) are usually implemented as a set of cooperating brokers each managing a set of local clients. The brokers are connected by overlay links, which are used to dispatch published notiﬁcations as well as issued and revoked subscriptions. For this purpose, each broker manages a routing table, which is used to forward incoming notiﬁcations to interested neighbor brokers and local clients. The employed routing algorithm determines the strategy by which routing tables are updated (e.g., in case of newly issued or canceled subscriptions) and has a major inﬂuence on the system’s eﬃciency. There is a general trade-oﬀ between the traﬃc needed by the routing algorithm to keep its tables up-to-date and the number of superﬂuous notiﬁcations that are forwarded unnecessarily when, otherwise, all notiﬁcations are simply ﬂooded in the network. Thus, depending on the client’s distribution and the dynamics of their interests and publication behavior, diﬀerent routing algorithms are better suited in diﬀerent situations. However, current publish/subscribe middleware implementations usually enforce the usage of a single, statically conﬁgured algorithm, which is employed system wide. Obviously, this leads to suboptimal results as distributions, interests, and publications may change over time and may also substantially diﬀer in distinct parts of the network. To overcome these limitations, we investigate hybrid routing conﬁgurations, which allow the usage of more than one routing algorithm within the network. This paper extends our previous work [10], presents a new formalism and evaluates it by simulations. Speciﬁcally, we deﬁne a novel class of hybrid routing schemes by determining constraints for valid conﬁgurations, which ensure a correct delivery of all notiﬁcations. Furthermore, it is shown how valid routing conﬁgurations can be transformed into each other—seamlessly during runtime— by subsequently switching algorithms on a link-by-link basis. Leveraging this technique, a heuristic is developed, which continuously adapts the routing conﬁguration to reduce the overall traﬃc. The presented algorithm is completely decentralized, requires local knowledge only, and makes the publish/subscribe system adaptive to changing distributions of clients, interests, and publications. The rest of the paper is structured as follows: Sect. 2 discusses related work. After giving some basics in Sect. 3, we introduce the new class of hybrid routing algorithms in Sect. 4. Then, in Sect. 5 we show how the reconﬁguration of the routing algorithm can be used to self-optimize the publish/subscribe system. Finally, we present an evaluation of our concept in Sect. 6 and present our conclusions in Sect. 7.

Self-optimizing Hybrid Routing in Publish/Subscribe Systems

2

113

Related Work

Using several routing algorithms in a single publish/subscribe system was ﬁrst suggested by Carzaniga [4]. He proposed to have clusters using hierarchical routing that are connected by an acyclic peer-to-peer protocol, where the structure of such a hybrid topology is derived manually from requirements in companies by the system administrators. Rendering publish/subscribe systems self-optimizing has already been approached in the past. Bittner and Hinze [1,2,3] introduced subscription pruning, which reduces the matching costs by replacing subscription predicates by more general predicates, which are simpler to evaluate. However, this potentially leads to increased costs for forwarding notiﬁcations since more notiﬁcations may be forwarded in which no client is interested in. To decide which predicates should be pruned, each broker determines a measure called selectivity degradation. Jaeger et al. [7] propose to adapt the interconnection of the brokers in a publish/subscribe overlay network to optimize the routing paths connecting publishers with interested subscribers. Since ﬁnding optimal overlays is NP-hard, a cost- and interest-driven heuristic only using local knowledge to determine reasonable edge substitutions is applied. The reconﬁgurations do not lead to service interruptions and multiple reconﬁgurations can be executed in parallel. Both approaches discussed above reveal valuable insights on how to design self-optimizing publish/subscribe systems. While our approach in principle pursuits the same objective, it dynamically adapts the applied routing algorithms to achieve cost reductions instead of modifying subscriptions or changing the overlay topology. Thus, our approach can be seen as orthogonal concept which may be combined with the others.

3

Routing Algorithms in Publish/Subscribe Systems

In this section we lay the foundation for further discussions by introducing the basic network model (Sect. 3.1) and two standard routing algorithms (Sect. 3.2), which we consider in this paper. 3.1

Network Model

A notiﬁcation service consists of a set of brokers B that are connected by an acyclic overlay. An overlay link between two brokers Bi , Bj ∈ B is described by two directed edges eij and eji , which are elements of the set of all directed edges E. A directed edge eij indicates the ﬂow of notiﬁcations from Bi to Bj and the reverse ﬂow of subscriptions from Bj to Bi . As depicted in Fig. 1, we deﬁne the set of predecessors Pij and the set of successors Sij of a certain edge eij as follows: Pij = {eki | eki ∈ E ∧ k = j} and Sij = {ejk | ejk ∈ E ∧ k = i}. The predecessor edges Pij connect neighbor brokers from which notiﬁcations are potentially received before they are forwarded via eij . Pij also describes the potential destinations of a subscription that is forwarded from Bj to Bi . On the other hand, successor edges Sij are potential destinations for notiﬁcations and potential sources of subscriptions of eij .

114

A. Schr¨ oter et al.

Notifications

Subscriptions

Fig. 1. Predecessors and successors of edge eij

3.2

Standard Routing Algorithms

A routing algorithm deﬁnes how routing tables are updated in reaction to issued and canceled subscriptions and how notiﬁcations are forwarded through the broker network. Typically, a single routing algorithm is applied in the whole overlay network and the applied algorithm is selected before the system is started. In this paper, we concentrate on two basic routing algorithms: flooding and simple routing. With flooding, no subscriptions are exchanged between brokers; instead, all notiﬁcations are ﬂooded into the overlay such that every broker receives every notiﬁcation. It is reasonable to use ﬂooding if the majority of brokers is interested in most of the notiﬁcations; otherwise it has a overly high notiﬁcation forwarding overhead. On the other hand, simple routing ﬂoods all subscriptions into the network such that every broker knows all subscriptions and can, thus, avoid sending notiﬁcations to neighbor brokers not needing them. Therefore, simple routing reduces the notiﬁcation overhead by introducing subscription overhead. There also exist more sophisticated routing algorithms, e.g., identity-based or covering-based routing [8], which avoid sending all subscriptions everywhere by exploiting similarities among the subscriptions. To keep the presentation compact, these algorithms are not considered in this paper. However, the presented approach can be extended to cover these algorithms.

4

Hybrid Routing Algorithms

The idea of hybrid routing algorithms stems from the analysis of hierarchical routing algorithms. In hierarchical routing, there is a dedicated root broker to which all notiﬁcations and subscriptions are ﬂooded. Downstream, notiﬁcations are ﬁltered by active subscriptions. On the other side, subscription messages are not sent down the overlay tree. This actually means that downstream ﬁltering algorithms, like simple routing, are used while upstream messages are ﬂooded. Beside hierarchical routing, there are a lot of possible combinations of routing algorithms within one overlay network. Consequentially, we now introduce the class of hybrid routing algorithms, which combines the usage of diﬀerent standard routing algorithms within one broker overlay network by using diﬀerent algorithms on diﬀerent edges. Hybrid routing algorithms are a set of routing algorithms, whereby each individual algorithm is deﬁned by a configuration,

Self-optimizing Hybrid Routing in Publish/Subscribe Systems

115

which is described in the following Sect. 4.1. Then, in Sect. 4.2 we describe how to switch from one conﬁguration to another without service interruption. 4.1

Configuration of Hybrid Routing Algorithms

A configuration R = {rij } deﬁnes which standard routing algorithm is used on which edge. The edge configuration rij ∈ {SR, F L} deﬁnes whether ﬂooding (FL) or simple routing (SR) is used on eij . If rij = F L, broker Bi forwards all notiﬁcations to Bj and Bj does not send any subscriptions to Bi . If rij = SR, Bj sends all subscriptions it receives from other brokers to Bi , where they are used to ﬁlter notiﬁcations that Bi is forwarding to Bj . In this way, the global nature of the standard routing algorithm is reduced to single edges. It is not possible to set all edge conﬁgurations independently because not all conﬁgurations guarantee a correct notiﬁcation service where no notiﬁcations are lost or duplicated. In the following, we derive which relation between the edge conﬁgurations rij must hold to ensure that the resulting hybrid routing algorithm works correctly. We call such a conﬁguration a correct routing configuration. The crucial issue for a correct routing conﬁguration is subscription forwarding. If one edge uses simple routing, it has to receive all necessary subscriptions for matching incoming notiﬁcations. This can be expressed by the following condition: ∀(ei,j ∈ E) : rij = SR → ∀(ek,l ∈ Sij ) : rkl = SR

(1)

The above equation means that the usage of simple routing on edge eij implies that all succeeding edges also use simple routing. This is necessary because the subscriptions originating from these edges are needed by broker Bi to forward all interesting notiﬁcations via eij to Bj . Homogeneous routing algorithms (cf. Sect. 3.2) are special cases of hybrid routing algorithms, where the edge conﬁgurations are the same for all edges. Obviously, homogeneous routing algorithms satisfy the condition mentioned above. Also, for hierarchical routing this can be shown easily. Apart from these algorithms, there are many other correct hybrid routing algorithms. 4.2

Reconfiguration of Hybrid Routing Algorithms

After deﬁning correct routing conﬁgurations, we now explain how to switch from one correct conﬁguration to another. With this mechanism, an administrator or an algorithm for self-optimization (introduced in Sect. 5) can manage the routing scheme in the overlay network at runtime. We denote a reconﬁguration of an edge eij by rij = [rc , rt ] with rc = rt and rc , rt ∈ {F L, SR}, where rc is the current conﬁguration and rt the target conﬁguration. During a reconfiguration, we want to guarantee the correctness of the notiﬁcation service, i.e., lost and duplicated notiﬁcations must be avoided. Following, we investigate how a reconﬁguration from ﬂooding to simple routing and vice versa can be realized.

116

A. Schr¨ oter et al.

Switching from Flooding to Simple Routing. A reconﬁguration from ﬂooding to simple routing rij = [F L, SR] is controlled by Bj . First, Bi must know which notiﬁcations Bj is interested in. Therefore, Bj forwards all active subscriptions that were generated by subscriptions received from successor edges Sij . Broker Bi uses these messages to build up a routing table, which represents the interests of Bj . After all active subscriptions have been forwarded, Bj sends a signal to Bi to start ﬁltering notiﬁcations before sending them to Bj . Additionally, Bj starts to forward all subscriptions and unsubscriptions to Bi , which it is receiving from other brokers. This completes the reconﬁguration to simple routing. This process implicitly assumes that Eq. 1 is satisﬁed. Since Bj must know the interests of its subtree for forwarding notiﬁcations to Bi , simple routing must be used downstream. In the case that Bj is a leaf broker, Sij is an empty set and Bj can, thus, always switch its incoming edge to simple routing. Figure 2 shows as an example the process of a complete reconﬁguration from homogeneous ﬂooding to homogeneous simple routing. The ﬁgure depicts a broker overlay consisting of brokers B1 . . . B5 and their connecting edges. To illustrate the process of reconﬁguration, we added a label to every edge. The labels S1 to S3 express the causal relation between the edge reconﬁgurations. S2 directly depends on S1, and S3 directly depends on S2. The ﬁrst possible reconﬁgurations are, thus, r3,1 = [F L, SR], r3,2 = [F L, SR] and r4,5 = [F L, SR], because B1 , B2 , and B5 are leaf brokers with an empty set Sij . All other edges have to apply ﬂooding because the criterion of Eq. 1 is not fulﬁlled. If B1 , B2 and B5 have ﬁnished their reconﬁguration (indicated by S1), the process is continued with r4,3 = [F L, SR] and r3,4 = [F L, SR] expressed by S2. This step can be performed because all edges in S3,4 and S4,3 are already using simple routing. Step S3 completes the reconﬁguration. Now all edges are using simple routing. Switching from Simple Routing to Flooding. Switching from simple routing to ﬂooding rij = [SR, F L] is controlled by Bi , which ﬁrst starts to ﬂood notiﬁcations via eij instead of matching them. Additionally, Bi drops its routing table and stops the processing of subscriptions that Bj is still sending. In the second step, Bi sends a message to Bj that it should stop sending subscriptions. With the processing of this message at Bj , the reconﬁguration from simple routing to ﬂooding is completed.

s1 s3 s1

s3

s2

s1 s2

s3

Fig. 2. Reconﬁguration from Flooding to Simple Rooting

Self-optimizing Hybrid Routing in Publish/Subscribe Systems

117

To avoid that rij = [SR, F L] leads to a violation of Eq. 1 by any predecessor edges Pij , it is necessary that all of these edges use ﬂooding. Since Pij is empty for leaf brokers, they can always switch outgoing edges to ﬂooding. For the example presented in Fig. 2, the switching from homogeneous simple routing to homogeneous ﬂooding would be executed just in the reverse order.

5

Self-optimizing Routing

Based on the results of Sect. 4, we are now able to use the reconﬁguration mechanism to step-wise reduce the costs of routing. We ﬁrst introduce a cost model in Sect. 5.1 and then describe how each broker can detect whether a reconﬁguration of an edge is proﬁtable or not in Sect. 5.2. In Sect. 5.3, we present the algorithm, which implements the self-optimizing routing. 5.1

Metrics

For optimization purposes we deﬁne a cost measure that directly depends on the N S notification rate ωij and the subscription rate ωij in the system at one moment 1 in time . An obvious approach for a cost measure Cij for edge eij is: N S Cij = cij · ωij + cji · ωij

(2)

The cost coeﬃcients cij and cji represent the costs for forwarding and processing one message. Depending on the optimization objectives, like optimizing network performance or load balancing, cij can be derived from metrics like CPU time, memory consumption, or forwarding latency. To derive the global cost measure, we simply sum the costs of all edges: C= Cij (3) ∀eij ∈E

The self-optimizing routing tries to reduce this cost measure by adapting the routing algorithm. To achieve this, beside the current costs Cij on one edge ij after a reconﬁguration of the routing algorithm are also the expected costs C important. Similar to Eq. 2 one can deﬁne them by: N S ij = cij · ω C ij + cji · ω ij

(4)

N S How these expected message rates of notiﬁcations ω ij and subscriptions ω ij can be determined is discussed in the following section.

5.2

Local Decisions

Now, we derive under which conditions the edge reconﬁgurations rij = [F L, SR] respectively rij = [SR, F L] reduce the costs of edge eij according to the cost 1

Subscriptions are forwarded from Bj to Bi .

118

A. Schr¨ oter et al.

model introduced before. To decide whether a reconﬁguration is reasonable or ij must be compared to the not, the expected costs after a reconﬁguration C current costs Cij . It is obvious that if Cij < Cij holds the cost of the link can be reduced by a reconﬁguration. This implies the reduction of the global costs of the system because there is no inﬂuence on other edges in the system (cf. Sect. 5.3). Since the current costs Cij can easily be determined by both brokers Bi and Bj , it is necessary to analyze which broker can derive the expected costs ij for both directions of reconﬁguration. C Flooding to Simple Routing. While using ﬂooding on edge eij , broker Bi sends all notiﬁcations to Bj without considering Bj ’s interests. This can lead to a high amount of notiﬁcations that Bj has to discard because neither Bj itself or one of its neighbors connected by Sij are interested in those notiﬁcations. If the number of discarded messages exceeds a certain level, it is reasonable to reconﬁgure the edge conﬁguration to simple routing, i.e., to send subscriptions to Bi and therefore to reduce the number of notiﬁcations. ij that will occur in simple routing, In order to compute the expected costs C N S the message rates ω ij and ω ij must be determined. Both values can be measured S by broker Bj . On one hand, the expected subscription message rate ω ij equals the sum of the local subscription rate and the aggregated message rates from N all successor edges Sij . On the other hand, the expected notiﬁcation rate ω ij can easily be derived since the broker is already matching notiﬁcations before forwarding them to neighbors. Thus, the expected rate is equal to the interest rate. Simple Routing to Flooding. If simple routing is applied on eij , broker Bj sends subscriptions to Bi that builds up a routing table to ﬁlter the notiﬁcations forwarded to Bj . If in some situation the eﬀort for using simple routing is higher than for ﬂooding, a reconﬁguration is suitable. This requires that Bj is interested in almost all notiﬁcations forwarded by Bi or that the amount of subscription messages is signiﬁcantly higher than the one of notiﬁcations. ij of using ﬂooding, it is only necessary to To compute the expected costs C N S determine the notiﬁcation message rate ω ij since ω ij = 0. The information about the expected ﬂooding rate is only available on broker Bi because it is receiving all notiﬁcations from Pij and of its local clients. This accumulated rate will appear on the overlay link eij if a reconﬁguration is triggered. Both reconﬁguration criteria can be evaluated by the broker that was the coordinator of a routing reconﬁguration before (cf. Sect. 4.2). Thus, it can be seamless integrated into an self-optimization algorithm, which is introduced next. 5.3

The Algorithm

If we combine all concepts introduced before, we can implement a self-optimizing algorithm consisting of three active steps that are periodically repeated:

Self-optimizing Hybrid Routing in Publish/Subscribe Systems

119

1. Each broker checks if there are candidate edges that can potentially be reconﬁgured. Depending on the current routing algorithm, for such an edge eij either all edges in Sij must use simple routing or all edges in Pij must use ﬂooding. 2. For each candidate edge, the broker checks if the reconﬁguration is reasonable. This is true if the expected costs are lower than the current costs. To avoid oscillation in the optimization process, a reconﬁguration is only triggered if the expected gain is higher than a predeﬁned value (e.g. 5%). 3. For each reasonable reconﬁguration, the reconﬁguration process is triggered as described in Sect 4.2. Beside the described active role, every broker implements listeners for the additional messages, which have been introduced for the reconﬁguration process (Sect. 4.2). These listeners directly implement the two actions “stop sending subscriptions” and “start ﬁltering”. Furthermore, incoming and outgoing message traﬃc is monitored in order to update the message rates for cost calculations. To reduce noise, these rates are calculated by averaging over a bounded time window. Major Properties. The described algorithm has the following major properties: (i) It is completely distributed, (ii) it requires only local knowledge, (iii) it does not depend on a complex interaction scheme, and (iv) it introduces only little additional traﬃc. While the ﬁrst property is trivial, the second one can directly be derived from the discussion of local optimizations in Sect. 5.2. Properties (iii) and (iv) result from the described reconﬁguration and optimization process in Sect. 4.2. There, we showed that any reconﬁguration is processed by ﬁrst evaluating the reconﬁguration condition, second by changing the local behavior, and third by sending a signal to neighboring brokers. Besides these signaling messages, the only additional messages that have to be sent are the messages required to build up the routing table when switching an edge from ﬂooding to simple routing. If the time between consecutive reconﬁgurations from ﬂooding to simple routing is not shorter than the average subscription lifetime, no additional messages compared to simple routing are needed. Furthermore, there is no need of additional coordination since reconﬁgurations are done edge by edge and there is no possible conﬁguration, where neighbor brokers will reconﬁgure the same edge at the same time: If ﬂooding is active on an edge eij , only broker Bj is able to trigger a reconﬁguration. If simple routing is used, only Bi is able to initialize a reconﬁguration. Therefore, the self-optimizing routing algorithm assures a conﬂict-free adaptation of the system. Locality. Interestingly, the reconﬁguration of an edge does not inﬂuence the message traﬃc on other edges. Based on Sect. 4.2, a reconﬁguration aﬀects only the rates of notiﬁcations and subscriptions on an individual edge. Therefore, cost reduction on a particular edge results in reduced global costs, too. This can be shown by examining a reconﬁguration rij = [F L, SR]. We know that all edges in

120

A. Schr¨ oter et al.

Sij use simple routing and all edges in Pij use ﬂooding, otherwise the correctness criterion deﬁned by Eq. 1 would be violated. In between, eij switches to simple routing and subscriptions are sent to Bi , but they are not forwarded to the edges in Pij because they use ﬂooding. On the other side, Bi ﬁlters incoming notiﬁcations before forwarding them to Bj , but there is no notiﬁcation that appears additionally on edges in Sij because Bi ﬁlters them already. These two facts show that rij = [F L, SR] does not inﬂuence other edge costs. Similarly, we can argue for rij = [SR, F L]. Optimality. In a static situation, all rates stay constant and local optimizations are repeated until no further reconﬁgurations are proﬁtable. The optimization process leads to costs that are equal to or less than that of homogeneous routing. Because of the locality mentioned before it is obvious that the global cost measure can be reduced in each step of self-optimization. Thus, local optimality is guaranteed. Furthermore, it can be shown that also global optimality can be reached if certain constraints for the cost coeﬃcients cij hold. Due to limited space, we cannot present the proofs here.

6

Evaluation

Now, we present the results of a discrete event-simulation of the proposed algorithm. The setup is based on a complete binary overlay tree with 127 brokers. There are 1, 000 diﬀerent types of notiﬁcations that clients can publish or exclusively subscribe to. The type and the originating broker of publications or subscriptions are randomly chosen. The expected overall publication rate is set to 500s−1. The initial subscription birth rate is set to 10s−1 with an expected subscription lifetime of one minute. All three processes are stochastic and exhibit exponential distributions. All cost coeﬃcients cij were set to 1 and the active role of the self-optimizing routing (Sect. 5.3) is called every 10 seconds. A reconﬁguration is only triggered if the expected gain is higher than 5%. The results have been averaged over 1, 000 runs. Figure 3 compares the global message traﬃc when using self-optimizing routing, homogeneous simple routing and homogeneous ﬂooding. In order to show the adaptiveness of self-optimizing routing, we modify the behavior of subscribers as follows: After 400 seconds, the subscription birth rate is continuously increased up to 20, 000s−1. This rate is reached approximately 850 seconds after simulation start. After a short period with a stable rate, it is then continuously reduced to 2, 000s−1. This value is reached at about 1800 seconds. During the phase of a low subscription birth rate, self-optimizing routing and simple routing show the same behavior because both use simple routing on all edges. In contrast to the behavior of simple routing, ﬂooding indicates a constant message traﬃc. Due to the property of ﬂooding, all notiﬁcations are routed in all possible directions. Therefore, ﬂooding is independent of the subscription birth rate. Although ﬂooding shows a bad performance in order to deliver all messages in the system, simple routing gets even worse in case that the subscription birth

Self-optimizing Hybrid Routing in Publish/Subscribe Systems

121

Fig. 3. Self-optimizing routing compared to simple routing and ﬂooding

rate exceeds a certain threshold. Approximately, after 800 seconds the curve indicating simple routing intersects the one for ﬂooding. As a matter of fact, the curve representing the message traﬃc in self-optimizing routing resides always under the curve for simple routing and ﬂooding. Due to its adaptive behavior, the algorithm reacts to the increase of the subscription birth rate by switching edge-wise beginning with the leaf brokers from simple routing to ﬂooding. One may notice that the curve for reconﬁgurations increases strongly between 400 and 600 seconds. In this period, the algorithm detects high subscription rates on several edges and switches them to ﬂooding. In total, the algorithm switched 140 out of 252 edges to ﬂooding in order to reduce the message traﬃc. All remaining edges still use simple routing. A reverse reconﬁguration phase can be seen after 1200 seconds when the subscription birth rate is reduced. Now some of the edges switch back to simple routing to avoid unnecessary delivery of notiﬁcations.

7

Conclusions

In this paper, we presented an optimization algorithm, which continually adapts the routing conﬁguration of a content-based publish/subscribe system to reduce its cost. The presented algorithm works decentralized with little traﬃc overhead and reduces the costs with respect to the introduced cost model. The optimization is based on the introduction of hybrid routing algorithms, which allow to combine several routing algorithms in one broker overlay network. Before introducing this approach, the same routing algorithm had to be applied by all brokers and it had to be selected before system start. Our approach is especially advantageous in dynamic scenarios, where the clients’ behavior varies over time. However, it is also useful in static scenarios, where the information necessary to determine the optimal routing algorithm is not available before runtime.

122

A. Schr¨ oter et al.

Although we used hybrid routing algorithms composed of simple routing and flooding as an example, our approach works with any combination of at least two routing schemes. Therefore, we want to extend the optimization algorithm to include other routing algorithms such as identity-based, covering-based, and merging-based routing. Additionally, it seems interesting to include advertisements, which are issued by producers to indicate the notiﬁcations they will potentially publish in the future. In this case, optimization will be applied on two layers corresponding to notiﬁcation and subscription routing. Despite the extensions in the ﬁeld of routing, our future work also addresses algorithmic challenges to avoid unnecessary reconﬁgurations even in highly dynamic situations. Therefore, the prediction of expected message rates and the consideration of reconﬁguration costs will be investigated.

References 1. Bittner, S., Hinze, A.: Dimension-based subscription pruning for publish/subscribe systems. In: Hinze, A., Pereira, J. (eds.) Proceedings of the 5th International Workshop on Distributed Event-Based Systems (DEBS 2006), Lisbon, Portugal, July 2006, pp. 25–25. IEEE, Los Alamitos (2006) 2. Bittner, S., Hinze, A.: Pruning Subscriptions in Distributed Publish/Subscribe Systems. In: Proceedings of the Twenty-Ninth Australasian Computer Science Conference (ACSC 2006), Hobart, Australia, January 2006. ACS (2006) 3. Bittner, S., Hinze, A.: Subscription Tree Pruning: A Structure-Independent Routing Optimization for General-Purpose Publish/Subscribe Systems. Technical Report 01/2006, Computer Science Department, University of Waikato (January 2006) 4. Carzaniga, A.: Architectures for an Event Notiﬁcation Service Scalable to Widearea Networks. PhD thesis, Politecnico di Milano, Milano, Italy (December 1998) 5. Eugster, P., Felber, P., Guerraoui, R., Kermarrec, A.-M.: The many faces of publish/subscribe. ACM Computing Surveys 35(2), 114–131 (2003) 6. Fiege, L., M¨ uhl, G., Buchmann, A.: An architectural framework for electronic commerce applications. In: Proceedings of the 2001 Annual Conference of the German Computer Society (Informatik 2001), Viena, Austria, September 2001, pp. 928–938 (2001) 7. Jaeger, M.A., Parzyjegla, H., M¨ uhl, G., Herrmann, K.: Self-organizing broker topologies for publish/subscribe systems. In: Liebrock, L.M. (ed.) SAC 2007, pp. 543–550. ACM, New York (2007) 8. M¨ uhl, G., Fiege, L., Pietzuch, P.R.: Distributed Event-Based Systems. Springer, Heidelberg (2006) 9. Pietzuch, P.R.: Hermes: A Scalable Event-Based Middleware. PhD thesis, Computer Laboratory, Queens’ College, University of Cambridge (February 2004) 10. Schr¨ oter, A., M¨ uhl, G., Richling, J., Parzyjegla, H.: Adaptive routing in publish/subscribe systems using hybrid routing algorithms. In: ARM 2008: Proceedings of the 7th workshop on Reﬂective and adaptive middleware, pp. 51–52. ACM, New York (2008)

Workload Management in Dynamic IT Service Delivery Organizations Yixin Diao and Aliza Heching IBM T.J. Watson Research Center P.O. Box 704, Yorktown Heights, NY 10598, USA {diao,ahechi}@us.ibm.com

Abstract. Enterprises and service providers are increasingly looking to global service delivery as a means for containing costs while improving the quality of service delivery. However, it is often diﬃcult to eﬀectively manage the conﬂicting needs associated with dynamic customer workload, strict service level constraints, and eﬃcient service personnel organization. In this paper we propose a dynamic approach for workload and personnel management, where organization of personnel is dynamically adjusted based upon diﬀerences between observed and target service level metrics. Our approach consists of constructing a dynamic service delivery organization and developing a feedback control mechanism for dynamic workload management. We demonstrate the eﬀectiveness of the proposed approach in an IT incident management example designed based on a large service delivery environment handling more than ten thousand service requests over a period of six months.

1

Introduction

Information Technology (IT) Services Delivery involves managing IT systems and processes to support business functions and delivering high quality IT services to meet customer needs. As deﬁned by IT Information Library (ITIL), IT Service Management consists of a cohesive set of best practices, such as the Service Desk for contact between service providers and service users, the Incident Management process to quickly restore normal service operations in the event of failure, and the Service Level Management process to ensure that Service Level Agreements (SLAs) are met with minimum impact on service quality [1]. An IT Service Delivery provider can manage its own IT environment or an outsourced environment supporting IT infrastructure for one or many customers. Outsourcing or multisourcing was originally driven only by cost considerations and attempts to focus on core competencies. There is increasing recognition that IT service providers can also ensure greater resiliency in case of disaster, business continuity due to time zone diﬀerences, and establishment of world wide best practices. In this paper we consider the Incident Management process, a signiﬁcant IT Service Management processes that requires a large number of service personnel. ITIL deﬁnes an incident as any event that is not part of standard operations and C. Bartolini and L.P. Gaspary (Eds.): DSOM 2009, LNCS 5841, pp. 123–137, 2009. c IFIP International Federation for Information Processing 2009

124

Y. Diao and A. Heching

results in a service interruption or reduction in service quality. Without lose of generality, we use the term “service request” to refer to the three categories of incidents, namely, application (e.g., disk usage threshold exceeded), hardware (e.g., system down), and service requests (e.g., forgotten password). Generally, service requests are associated with service level agreements that are designed at the time of contract negotiation and specify the target service response time. The objective of the IT service delivery provider is to most cost eﬀectively meet the service level agreements. However, managing the service delivery organization and meeting this objective is challenging: customer workload contains inherent uncertainty, new customer accounts are onboarded frequently, established service requests may be routed to diﬀerent service delivery environments for lower cost delivery, and the so-called steady state simply does not exist. The majority of the existing literature in the area of service request workload management is conﬁned to change requests, which can be scheduled in advance. [14] introduces a system for automated change management that accounts for precedence constraints. [18] solves the change scheduling problem by using a business-driven approach that evaluates change schedules in terms of the ﬁnancial loss. [21] proposes a change scheduling optimization model that can be solved using standard mathematical programming techniques. More relevant is the work presented in [5] and [6] which develop a simulation model to guide the workforce decision making in IT incident management. With respect to workforce management, [13] considers the problem of long term workforce planning with general nonstationary arrival and service time processes. [20] studies dynamic staﬃng in a call center environment where the objective is high service level attainment. [7] proposes a method for determining the optimal numbers of permanent versus temporary staﬀ and the threshold value at which temporary staﬀ should be called upon, considering conﬂicting objectives of meeting service level constraints and minimizing costs. Although these papers include aspects of managing the dynamic service delivery environment, all of them depend on accurate models (which are diﬃcult to obtain in real world systems). A class of workload management methods based on control theory provides an alternative framework for studying the behavior of dynamic systems and feedback-driven control [12]. Various control-theoretic approaches to performance management of web server, diﬀerentiated caching, and multimedia streaming have been studied. [2] proposes an admission control method using classical feedback control theory and demonstrates this method for maintaining CPU utilization of an Apache web server. In [8] both CPU and memory utilizations are managed by a multiple-input, multiple-output controller that manipulates interrelated tuning parameters such as MaxClients and KeepAlive. In [17] an adaptive resource controller is developed to adjust resource shares and meet application-level quality of service goals in a data center environment. However, none of the above feedback approaches has been applied to workload management in service delivery organizations. In this paper we propose a feedback control approach for workload management.The approach dynamically adjusts organization of service delivery personnel

Workload Management in Dynamic IT Service Delivery Organizations

125

based upon the observed gaps between measured and target service level metrics, in response to the changes in the business environment. The proposed approach does not require detailed system models. Rather, it employs feedback control to build closed loop systems that sense and correct the errors in modeling and control as well as changes in workloads and resources. In addition, compared to existing feedback control approaches used in systems management, the proposed approach diﬀerentiates itself as (i) devising a means to decompose the combined service level optimization problem into a set of single input single output control problems with speciﬁc service operation targets as reference inputs, and (ii) proposing an uncertainty-based adaptive control approach that updates the control parameters without detailed system models (which are typically diﬃcult to obtain in a service delivery environment). The remainder of this paper is organized as follows. Section 2 discusses the workload management problem in IT service delivery organizations. Section 3 presents the architecture and algorithm of a feedback control mechanism and uncertainty-based learning and adaption. Section 4 describes the controller evaluation results in an IT incident management example built using data from a large service delivery organization. Our conclusions are contained in Section 5.

2

Service Delivery Workload Management

In this section we describe the workload management problem in IT service delivery organizations. Recognizing that the real world scenario is full of details which typically render the problem highly complex and intractable; in the interest of simplicity and applicability, we refrain from modeling every aspect of incidents and service organizations. Instead, we focus on the key properties and ﬁrst order eﬀects. Figure 1 illustrates the process and operation ﬂow of a service delivery environment. Customers interact with the service delivery environment through the service request management system, which coordinates service request creation, queueing, assignment, and closure. Based on the nature of the required services, service requests are assigned to diﬀerent service delivery units which are comprised of a group of service agents (human resources) providing a set of services to the customers. Typically, the arrival process of service requests are independent, although a batch of requests may arrive as a result of a major failure (e.g., database or network failure) that aﬀects multiple components within an IT infrastructure. Generally, a service request only requires service from an individual service agent and can be resolved within several hours. As such, we do not consider change requests (that require scheduling to ﬁt within predeﬁned change windows) or project requests (that require signiﬁcant coordination between the service agents and customer); both of these will have a service response time lasting from several days to several months. We also do not consider mis-routed service requests, since handling and improving such requests is out of the scope of workload and personnel management. A service delivery environment is generally organized as a set of service delivery units, each servicing a subset of customers and consisting of service agents

126

Y. Diao and A. Heching Service Delivery Environment

Service Delivery Unit

Customer

Service Delivery Unit Customer

… Customer

Service Request Management System

Service Delivery Unit

… Service Delivery Unit

Fig. 1. Illustrative process and operation ﬂow of a service delivery environment

with similar skills. Assigning groups of customers to one or more service delivery units allows service agents to develop familiarity with customers’ IT infrastructure, enabling improved quality of service. In addition, service agents are assigned to speciﬁc service delivery units based on skills, to achieve cost eﬃciencies; less complex requests can be routed to lower skilled units, while agents from the higher skilled units can focus on more complicated service requests. One challenge faced by service delivery organizations is to determine the optimal size of a service delivery unit. An oversized service delivery unit results in low staﬀ utilization, while an undersized service delivery unit may have diﬃculty achieving service level targets. Furthermore, the customer workload may vary over time. Although some variations may be predictable (for example, the banking industry may have fewer service requests at the end of the year when the ﬁnancial activity slows down), other variations may be driven by random factors. The changes in customer behaviors directly impact the workload received by the service delivery units, and thus impacts their abilities to meet the service level agreements. Next, we describe the architecture of the dynamic workload management system, where service agents are dynamically reassigned to service delivery units in response to changes in customer workload. The objective of the dynamic assignment is to balance service level attainment across service delivery units. Typically, there is a nonnegligible time penalty associated with human resource reassignment. In this paper, we assume this penalty is negligible by taking into account that the service delivery units are colocated and that reassignment only occurs between adjacent service units, where adjacency is measured by skills required to resolve the service request (customer familiarity and technical knowledge). For similar reasons, we do not consider a hiring/ﬁring approach to workforce size adjustment, since this is costly and involves time consuming activities such as training and knowledge transer. Figure 2 depicts the architecture. The feedback controller assigns service agents to the service delivery units, which handle customer service requests. Service requests are associated with service level targets that are customer speciﬁc. Service

Workload Management in Dynamic IT Service Delivery Organizations

127

Service Level Targets Customers

Service Operation Target Service Objective Alignment

Service Agent Assignment Feedback Controller

Service Requests

ServiceDelivery DeliveryUnit Unit Service Service Delivery Units

Service Measurement

Control Parameters Adaptive Design

Service Level Metrics

Fig. 2. Architecture of dynamic workload management

level metrics are periodically measured from the service delivery units, compared to the service level targets, and used by the feedback controller to adjust service agent assignment, if needed. We now discuss in greater detail the three key modules of the dynamic workload management scheme: service objective alignment, feedback controller, and adaptive design. The goal of the service objective alignment module is to align the service level targets from the customers with the measured service level metrics from the service delivery environment, in order to create the service operation targets for the service delivery units. There are several distinctions here. First, the service operation target of the service delivery unit is created based on customers’ business objectives regarding service level attainment (or average request resolution time, if speciﬁed by the customers), rather than internal operational metrics such as agent utilization or backlog length. Second, the service operation target is not derived based solely upon the service level targets at the service design time, but considers the actual measured service level metrics (which may vary as customer workload changes). Third, while the service requests from multiple customers are being serviced by multiple service delivery units, the purpose of aligning service level metrics with service level targets is to ensure fairness among customers. That is, we leverage the concept of load balancing. The management objective is either to meet the service level targets equally well (i.e., to keep the same safety margin between the targets and metrics in order to increase the service resiliency of unknown workload changes) or to equalize the partial derivatives of the service level metrics (which leads to a global optimal solution given convex cost functions [9]). This fairness objective can be augmented through weighting to consider diﬀerences, such as diﬀerent service level infringement penalties, between customers. The feedback controller operates to achieve the service operation target by dynamically adjusting the assignment of service agents to service delivery units. Indeed, to increase the scalability of dynamic workload management, we apply the separation principle to deﬁne the control objective centrally and operate the controller distributively. There is one controller for each service delivery unit, which focuses on a single metric (the service operation target deﬁned for that

128

Y. Diao and A. Heching

service delivery unit) and manipulates a single control (the required number of service agents for the unit). The collaboration across service delivery units occurs at a higher level: deﬁning the service operation targets through service objective alignment to achieve fairness, and constraining the total number of service agents due to resource constraints. The adaptive design module takes as input the operational data regarding service agent assignment and service level metrics, and outputs the control parameters for the feedback controller to provide appropriate control performance (i.e., quick response to workload changes while not overreacting to system noise). We note that the above mechanism follows the monitoring, analysis, planning, and execution (“MAPE”) model from autonomic computing. It supports decision making on dynamic workload and personnel management, and studies a simple feedback loop that captures the ﬁrst order eﬀect of adjusting service agents to improve service level attainment. The leverage that the feedback control mechanism provides is increased robustness of the solution in response to the stochasticity inherent in customer service request arrival patterns as well as service agent service times. In addition, as discussed in greater detail in the next section, the reactive nature of the system (i.e., the system feedback coupled with dynamic workload decision making) is designed to increase the likelihood that service level targets are met. Further, the simple yet eﬀective structure of the control mechanism reduces the dependency on complicated system models. In contrast, optimization approaches either require deterministic input or, in cases of stochastic optimization, can result in complex models that are challenging to solve. Heuristic approaches, on the other hand, may result in solutions that oscillate and are impractical to implement.

3

Uncertainty-Based Adaptive Control

Following the architecture discussion presented in the previous section, in this section we describe the implementation of the dynamic workload management system in more detail: (i) Service objective alignment. We construct the service operation targets from service level objectives and metrics, and decompose the control strategy to enable single-input single-output controllers. (ii) Feedback control rule. We adopt a simple and yet eﬀective control rule based on the Bang-Bang principle [19], and service agent assignment is adjusted based upon feedback from service level metrics. (iii) Adaptive design and operation. We present a set of methods that help to conﬁgure control parameters with minimal modeling eﬀort. 3.1

Service Objective Alignment

Feedback controllers implement goal-driven algorithms to track a reference signal and adjust the control input with the objective of minimizing the error between the measured metric and the target reference. In this section we describe how feedback controllers can be used to achieve fairness among a set of service level

Workload Management in Dynamic IT Service Delivery Organizations

129

targets from multiple customers. The key method is to decompose the combined targets so that they can be fulﬁlled by multiple single input single output controllers. Consider a service delivery environment that services M customers from N service delivery units. For each service delivery unit i, i = 1, . . . , N , we deﬁne a service operation performance function fi (SLi (k), SL∗i ) =

M

wm SLi,m (k) − SL∗i,m

(1)

m=1

where SL∗i,m denotes the service level target for service requests serviced by service delivery unit i and originating from customer m, SLi,m (k) denotes the measured service level metrics at time interval k, and wm indicates the weighting factor for customer m. The service level target SL∗i,m could be the service level attainment target, e.g., percentage of service requests serviced within a predeﬁned time interval or average service request completion time. The controller is designed to achieve “fairness” among all service delivery units, that is, eliminating the diﬀerence between performance functions fi (SLi (k), SL∗i ) = fj (SLj (k), SL∗j )

(2)

for all pairs of service delivery units i and j. Namely, the controller attempts to achieve balanced service levels attainment across all service delivery units. While Equation (1) deﬁnes such balance in terms of the service level attainment “margin,” other types of performance functions can be used to characterize SLA violation penalties as well [3]. As described in the Performance Pyramid System developed by Lynch and Cross [16], metrics are used to measure an organization’s performance on market and ﬁnancial objectives. Together, these two types of metrics drive an organization to achieve the overall corporate vision. Metrics that are aligned with market objectives measure external eﬀectiveness; metrics that are aligned with ﬁnancial objectives measure internal eﬃciencies. In this paper we focus on a market performance metric – service level attainment. As it is a measure of customer satisfaction and lies in the group of external eﬀectiveness measures, we consider balanced service level attainment. Other metrics that consider the ﬁnancial objectives of the organization would consider factors such as diﬀering penalties for service level violations. From the feedback control perspective, this fairness objective can be achieved by constructing a set of single input single output controllers, one for each service delivery unit, with the common reference signal deﬁned as N 1 fj (SLj (k), SL∗j ) N j=1

(3)

and the control error for controller i as ei (k) = fi (SLi (k), SL∗i ) −

N 1 fj (SLj (k), SL∗j ) N j=1

(4)

130

Y. Diao and A. Heching 1ui(k)

si

di

-di

ei(k)

-si

Fig. 3. An illustrative of Bang-Bang Control Law

Thus, the objective of the feedback controller is to drive the control error ei (k) to zero using the following feedback control law Δui (k) = u(k + 1) − u(k) = Ki (ei (k))

(5)

where ui (k) denotes the control input (the number of service agents in service delivery unit i). Control law (5) looks similar to the integral control law that is used in proportional-integral-derivative (PID) controllers [12]. Although both forms aim to eliminate steady-state errors, the diﬀerence is that Ki (·) represents an integral function instead of an integral gain, as in the PID controller. This increases the design ﬂexibility, as we will discuss in the following sections, so that we can design the controller directly based on system behaviors and uncertainties instead of building diﬀerence equation models through system identiﬁcation. 3.2

Feedback Control Rule

Although a rich set of controllers has been studied in control literature, the majority are model-based and require extensive modeling and model-based adaptation [4,11]. In the previous section we decomposed the multiple customer service level attainment problem into a single input single output control problem. In this section we leverage a simple yet eﬀective control rule based on the BangBang logic. A Bang-Bang (on-oﬀ) controller is frequently used in optimal control where the control input is restricted between an upper bound and a lower bound, and an optimal solution is to switch between the control bounds [10]. We apply the Bang-Bang controller due to the coarse granularity of service agents (i.e., they can only be moved in increments of full individuals) and the restricted size of the service delivery units. This makes an on-oﬀ controller more applicable, compared to other continuous or discrete control laws. We also enhance the on-oﬀ controller with the integral control rule, as in Equation (5), and a dead zone to ensure system stability. Figure 3 depicts the operation of the Bang-Bang control law for controller i. The x-axis denotes the control error ei (k) and the y-axis denotes the change in control input Δui (k). A dead zone is deﬁned so that no control input adjustment is given if the control error is

Workload Management in Dynamic IT Service Delivery Organizations

131

between −di and di . If the control error exceeds the dead zone, the control input is adjusted by step size si . In the case of a positive error (that is, service delivery unit i performs worse than average and performs worse than all service delivery units) the Bang-Bang control law will increase the control input (i.e., add service agents) to reduce the deviation. When control input dependency exists (e.g., the number of service agents assigned to each delivery unit must be positive and total number of service agents assigned to all delivery units must equal the total number of service agents available) a projection algorithm can be applied. Based on the control interval, the Bang-Bang controller operates periodically to drive the control error within the dead zone. Once in the dead zone, the controller monitors the control errors but no adjustments are made to the control input until changes in workload (e.g., increase in service request volume) drive the control error out of the dead zone. Overall, the performance and eﬀectiveness of the Bang-Bang controller is determined by careful selection of the three control parameters: control interval (T ), deadzone size (di ), and step size (si ). 3.3

Adaptive Design and Operation

While Bang-Bang control is widely used in optimal control with rigorous analysis [10], in this section we propose a simple and yet eﬀective design approach that does not rely on detailed modeling and can react to system uncertainty arising from multiple sources. In a services delivery environment, sources of system uncertainty include: (i) System Randomness. Service requests are generated at random times and with diﬀerent levels of complexity. Further, much of the measured data in a service delivery environment depends upon accurate recordings from service agents. Experience has shown that this data does not always reﬂect real events with perfect precision, due to the discrete (and unpredictable) nature of human beings. (ii) Transient Dynamics. There are often lags between the time when events occur (e.g., the service agent moves to a diﬀerent service delivery unit) and the time when the metrics are measured (e.g., when the service level metric improvement is actually realized and reported after the agent settles down), introducing uncertainty as to the true state of the system during the transient phase. (iii) Workload Variation. Both system conﬁgurations (e.g., assignment of customer accounts to the service delivery units) and workload behaviors (e.g., service request arrival rate per customer, rate at which service agents handle service requests) can vary over time. Thus, robustness and adaptability are desired for a feedback controller that operates in a real environment. By using the Bang-Bang control logic and the three control parameters deﬁned above, we design the adaptive feedback controller to accommodate these sources of system uncertainty. Control Interval. The control interval determines how frequently to adjust the control input, that is, the service agent assignment. Frequent adjustments may cause instability; infrequent adjustments implies that the controller may not be suﬃciently reactive to workload changes.

132

Y. Diao and A. Heching

We determine the appropriate control interval from the perspective of understanding and managing system randomness. Intuitively, if data variability is high, a larger control (sample) interval is required to ensure meaningful service measurement and control. According to the Central Limit Theorem, the distribution of the sample average of random variables approaches the normal distribution with a mean equal to that of the parent distribution and variance equal to that of the parent divided by the sample size (N ), irrespective of the parent distribution. For an initial control interval T0 , we measure the service level metrics SLi (k) and calculate the mean μSLi and the standard deviation σSLi . Given the desired σ i noise ratio r = μSL from the control designer, we can compute the control SLi interval (T ) as follows. σSL ∗ σSL = rμSLi = √ i i Ni T = T0 max Ni

(6) (7)

Based on experience, we set r = 0.1 to balance the feedback controller between sensitivity to system randomness and ability to adapt to workload changes. Dead Zone Size. The dead zone is used to increase controller robustness to system randomness and workload variation. Since the impact of system randomness cannot be entirely eliminated using control interval selection, the dead zone is introduced to avoid control oscillation, especially around the optimal steady state when the control error appears small. A dead zone is also valuable when the control input has a coarse granularity. For example, service agents can only be reassigned in increments of full individuals, even if the theoretical optimal value indicates a fractional adjustment. We design the dead zone size as follows di = lμSLi

(8)

where l is the threshold limit that makes the dead zone size proportional to the average of the service level metric. Typically, we choose l = 2r, with the objective that no control action should be reacting to system randomness. Generally, a larger threshold limit can reduce oscillation but may also lead to larger steady state error. Step Size. The step size is related to the speed of controller convergence. A larger step size results in faster controller response regarding workload variation, but may cause the controller to oscillate around the optimal point with control error bouncing around the dead zone. Conversely, a smaller step size leads to longer convergence time. From our experience, we choose an initial step size s equal to 5% of the control range (the number of service agents in the service delivery unit). If the step size is too large and causes oscillation around the dead zone, we introduce an oscillation-induced adaptation algorithm to resize the dead zone, as

Workload Management in Dynamic IT Service Delivery Organizations

133

follows: (i) Observe the control input history and record the sign of control input change. (ii) If an oscillation pattern is detected (e.g., the number of increases is equal to the number of decreases, e.g., 1, -1, 1, -1), increase the dead zone size by 20%. (iii) If a chasing pattern is detected (e.g., 1, 1, 1, 1, or -1, -1, -1, -1), decrease the dead zone size by 20%. In addition to resizing the dead zone, this adaptation algorithm can be used to adjust the step size. The diﬀerence occurs when the granularity of the step size is too coarse. For example, adding or removing one service agent in a small service delivery unit may cause large control errors. In this case, we cannot further reduce the step size, but must increase the dead zone size to avoid oscillation.

4

Evaluation

In this section we illustrate how the proposed feedback controller can be used for workload management in IT service delivery organizations. Our evaluation is based on incident management data collected from a large service delivery environment over a period of six months, including more than ten thousand service requests. Each incident service request includes details about the incident arrival time, service time, and completion time. A separate data source provides the service level agreement information with target service response time and target service level attainment. We analyze the service requests in order to characterize the workload. (For the purposes of this evaluation, some workload parameters have been modiﬁed to ensure sensitive business data are not revealed.) Our objective is to build a service testbed (simulator) based on a set of queueing models (speciﬁcally, M/M/m models [15]), that are calibrated using the collected service request data. We use the model to represent the dynamic behavior of the service delivery environment and examine how the proposed controller can be used for dynamic service agent assignment. Table 1 lists the workload parameters for our evaluation, where the workload arrives from four customers. The inter-arrival times of the service requests follow exponential distributions, as approximated from the collected data. (We evaluated goodness-of-ﬁt using the Kolmogorov-Smirnov test where the test statistic, the least upper bound for the cumulative distribution function, is 0.17 for the exponential distribution, compared to, for example, 0.28 for the normal distribution, and 0.70 for the lognormal distribution.) Table 1 also summarizes the Table 1. Workload parameters from an IT incident management example

Inter-arrival time (min) Service Delivery Unit 1 (%) Service Delivery Unit 2 (%)

Customer 1 Customer 2 Customer 3 Customer 4 14.2 45.4 177.1 20.1 92.5 83.8 64.4 82.1 7.5 16.2 35.6 17.9

134

Y. Diao and A. Heching

Agents

20 Unit 1 Unit 2

15 10 5 0

2

4

6

8

10

12

14

16

18

20

Attainment

100 Unit 1 Unit 2

98 96 94 92 90

2

4

6

8

10

12

14

16

18

20

Interval

Fig. 4. Eﬀect of static service agent allocation based on a heuristic rule

workload distribution across the two service delivery units. In addition, the average service time per service request is 62.2 minutes in service delivery unit 1 and 123.9 minutes in service delivery unit 2. These service times follow exponential distributions as well. The target service response time is 8 hours for all service requests, and the service level attainment target is 95% for all customers in each of the service delivery units. (This is a simpliﬁcation used in this queueing model; in a separate eﬀort, we are building a discrete event simulation model for better representation). There is a total of 20 service agents in the two service delivery units. The control decision to be made in each control interval is the allocation of service agents to each of the service delivery units, with the objective of fairly meeting the service level attainment targets. Typically, there is a mismatch between the desires of the service designers and the levers they are provided. In a service delivery environment, service designers wish to meet service level attainment target (with a safety margin) on all the service delivery units. However, this can only be achieved indirectly by adjusting service agent allocation (or service request routing). A heuristic method for assigning service agents to service delivery units is based upon the ratio of arrival rates and service rates. As indicated in Table 1, the arrival rate for delivery unit 1 is 0.128 requests/minute, which is 6.4 times the arrival rate for delivery unit 2 (0.02 requests/minute). Further, the average service time per service request in delivery unit 1 is approximately half that of delivery unit 2. Given these rates, one may expect the workload in delivery unit 1 to be approximately 3.2 that of Unit 2. If delivery unit 1 is allocated approximately 3.2 times the number of service agents than delivery unit 2 (i.e., 15 service agents to delivery unit 1 and 5 to unit 2), one may expect that the workload will be balanced and the service level attainment target can be equally met in both delivery units. Although simple, this algebraic rule does not capture the nonlinearity inherent in the queueing system. To illustrate, we applied this heuristic rule and the results are displayed in Figure 4. The x axis shows the control (sample) interval, and each interval is 10 days for service level metrics measurement. The upper graph plots the number of service agents assigned to the two service delivery units, and the lower graph

Workload Management in Dynamic IT Service Delivery Organizations

Agents

10 0

2

4

6

8

10

12

14

16

18

95 90

Unit 1 Unit 2 2

4

6

8

10

12

14

16

18

Unit 1 Unit 2 10 0

20

100

20

90 4

2

2

−2 −4

2

4

6

8

10

12

14

16

18

20

10

15

20

25

30

35

40

Unit 1 Unit 2

95

4 0

5

100

Error

Error

Attainment

20 Unit 1 Unit 2

Attainment

Agents

20

135

5

10

15

5

10

15

20

25

30

35

40

20

25

30

35

40

0 −2 −4

Interval

Interval

(a) Unknown workload

(b) Workload change

Fig. 5. Control performance of the feedback controller

plots their service level attainment over time. One can observe the gap between the service level attainment between the two service delivery units. Moreover, the attainment metric for service delivery unit 2 is hovering about the 95% target, causing SLA violations. Alternatively, the service designer can conduct experiments to determine the desired agent allocation. These experiments are time-consuming and must be repeated with changes in workload. Figure 5(a) displays the performance of the proposed feedback control mechanism. The controller begins in the same static setting as that in Figure 4 and converges to a balanced service level attainment based on sensing the diﬀerence between service level attainment in the two service delivery units. The bottom ﬁgure displays the control error (the solid line) and the dead zone (the two dashed lines). As the control error enters the dead zone, no more service agent adjustment is performed. Figure 5(b) displays the controller performance when the workload changes at interval 10. Speciﬁcally, the workload distribution between service delivery units 1 and 2 changes to 62.5% and 37.5%, respectively, for customer 1. This could occur, for example, if customer service requests follow seasonal patterns, or if the service delivery provider changes the assignment of customers (or types of customer service requests) to service delivery units. The controller begins with the optimal setting from Figure 5(a). When the workload changes, service attainment increases for service delivery unit 1 since fewer service requests arrive, prompting the controller to shift service agents to service delivery unit 2. After measuring the the service level metrics, the controller modiﬁes the service agent allocation to nine service agents in service delivery unit 1 and eleven service agents in service delivery unit 2. Overall, the proposed control mechanism successfully balances service level attainment across service delivery units, both when the workload pattern is stationary over time and when the workload dynamically changes. Further, appropriate choice of step size combined with dead zone deﬁnition prevents the controller both from oscillating and from responding to system noise.

136

5

Y. Diao and A. Heching

Conclusions and Future Work

Enterprises and service providers are increasingly looking to global service delivery as a means for containing costs while improving the quality of service delivery. In this paper we proposed a feedback control approach to dynamically adjust personnel based on dynamically changing workload. We also demonstrated the eﬀectiveness of the proposed approach with an IT incident management example, designed using data from a large service delivery environment with more than ten thousand service requests over a period of six months. While the initial results are encouraging, there are several areas for further research. First, the work presented in this paper focuses on dynamic service agent allocation. The underlying assumption is that there is no productivity impact (or “switching costs”) associated with moving service agents between service delivery units. We assume that the service delivery units are colocated so that productivity impact would not come in the form of physical considerations such as required travel time or oﬃce space setup. However, other considerations include factors such as potential impacts on collaboration, team productivity, and communication between agents. Social network analysis may be useful to understand how service agents interact with other service agents within their service delivery unit and with service agents in other service delivery units as a measure of the impact of service agent switching. Second, while preserving the simplicity and applicability of the current approach, we can also extend the modeling considerations to include the service agent skills and the diﬀerence between service delivery units. Third, it is our intention to further validate our approach using the aforementioned discrete event simulation and study the impact of various workload mix and changes, and the eﬀectiveness of control decisions (regarding step size, deadzone size, and control interval) in the presence of multiple (three or more) service delivery units. Finally, besides dynamic service agent allocation, we would like to study optimal service request routing as well as optimal mappings of customers to service delivery units.

References 1. IT Infrastructure Library. ITIL Service Support, version 3. Oﬃce of Government Commerce (2007) 2. Abdelzaher, T.F., Lu, C.: Modeling and performance control of Internet servers. In: Proceedings of the 39th IEEE Conference on Decision and Control, Sydney, Australia, pp. 2234–2239 (2000) 3. Aib, I., Boutaba, R.: On leveraging policy-based management for maximizing business proﬁt. IEEE Transactions on Network and Service Management (2007) 4. Astrom, K.J., Wittenmark, B.: Adaptive Control, 2nd edn. Addison-Wesley Publishing Company, Reading (1994) 5. Bartolini, C., Stefanelli, C., Tortonesi, M.: SYMIAN: A simulation tool for the optimization of the IT incident management process. In: Proceedings of IFIP/IEEE International Workshop on Distributed Systems: Operations and Management (2008)

Workload Management in Dynamic IT Service Delivery Organizations

137

6. Bartolini, C., Stefanelli, C., Tortonesi, M.: Business-impact analysis and simulation of critical incidents in IT service management. In: Proceedings of IFIP/IEEE International Symposium on Integrated Network Management (2009) 7. Bhandari, A., Scheller-Wolf, A., Harchol-Balter, M.: An exact and eﬃcient algorithm for the constrained dynamic operator staﬃng problem for call centers. Management Science 54, 339–353 (2008) 8. Diao, Y., Gandhi, N., Hellerstein, J.L., Parekh, S., Tilbury, D.M.: Using MIMO feedback control to enforce policies for interrelated metrics with application to the Apache web server. In: Proceedings of Network Operations and Management (2002) 9. Diao, Y., Hellerstein, J.L., Storm, A., Surendra, M., Lightstone, S., Parekh, S., Garcia-Arellano, C.: Using MIMO linear control for load balancing in computing systems. In: Proceedings of the American Control Conference, Boston, MA (2004) 10. Fleming, W.H., Rishel, R.W.: Deterministic and Stochastic Optimal Control. Springer, Heidelberg (1996) 11. Franklin, G.F., Powell, J.D., Emani-Naeini, A.: Feedback Control of Dynamic Systems, 3rd edn. Addison-Wesley, Reading (1994) 12. Hellerstein, J.L., Diao, Y., Parekh, S., Tilbury, D.M.: Feedback Control of Computing Systems. John Wiley and Sons, Chichester (2004) 13. Jennnings, O.B., Mandelbaum, A., Massey, W.A., Whitt, W.: Server staﬃng to meet time-varying demand. Management Science 42, 1383–1394 (1996) 14. Keller, A., Hellerstein, J., Wolf, J., Wu, K., Krishnan, V.: The CHAMPS system: Change management with planning and scheduling. In: Proceedings of IFIP/IEEE Network Operations and Management Symposium, Seoul, Korea, pp. 395–408 (2004) 15. Lavenberg, S.S. (ed.): Computer performance modeling handbook. Academic Press, Orlando (1983) 16. Lynch, R.L., Cross, K.F.: Measure Up! How to Measure Corporate Performance. Blackwell Pub,. Malden (1995) 17. Padala, P., Shin, K.G., Zhu, X., Uysal, M., Wang, Z., Singhal, S., Merchant, A., Salem, K.: Adaptive control of virtualized resources in utility computing environments. In: Proceedings of the 2007 EuroSys Conference, Lisbon, Portugal, Lisbon, Portugal (2007) 18. Reboucas, R., Sauve, J., Moura, A., Bartolini, C., Trastour, D.: A decision support tool to optimize scheduling of IT changes. In: Proceedings of IFIP/IEEE International Symposium on Integrated Network Management, Munich, Germany (2007) 19. Sonneborn, L.M., Van Vleck, F.S.: The bang-bang principle for linear control systems. SIAM Journal on Control and Optimization, 151–159 (1964) 20. Whitt, W.: Dynamic staﬃng in a telephone call center aiming to immediately answer all calls. Operations Research Letters 24, 205–212 (1999) 21. Zia, L., Diao, Y., Rosu, D., Ward, C., Bhattacharya, K.: Optimizing change request scheduling in IT service management. In: Proceedings of IEEE International Conference on Services Computing (2008)

CHANGEADVISOR: A Solution to Support Alignment of IT Change Design with Business Objectives/Constraints Roben Castagna Lunardi1, Weverton Luis da Costa Cordeiro1, Fabrício Girardi Andreis1, Juliano Araujo Wickboldt1, Cristiano Bonato Both1, Luciano Paschoal Gaspary1, Lisandro Zambenedetti Granville1, David Trastour2, and Claudio Bartolini3 1

Institute of Informatics, Federal University of Rio Grande do Sul, Brazil 2 HP Laboratories Bristol, UK 3 HP Laboratories Palo Alto, USA {rclunardi,weverton.cordeiro,fgandreis,jwickboldt, cbboth,paschoal,granville}@inf.ufrgs.br, {david.trastour,claudio.bartolini}@hp.com

Abstract. Change planning represents a key element for the operation and management of Information Technology infrastructures and services. Its scope ranges from the high level design of a change request to the generation, either manually or automatically, of detailed plans that, if executed, will perform the requested changes (e.g., modification of network device settings and deployment of new services). A fundamental problem is that, although correct, such detailed plans may not be necessarily aligned with the requirements defined in the business level (e.g., minimization of the downtime of a given service). To overcome this problem, in this paper we propose a solution for the alignment of change plans with business objectives/constraints. The solution is analyzed experimentally through a prototypical implementation of a decision support system called CHANGEADVISOR, which helps operators to understand the trade-offs between alternative change designs. Keywords: IT service management, business-driven IT management, IT change management.

1 Introduction The increasing size and complexity of Information Technology (IT) infrastructures and services have demanded the adoption, by organizations, of best practices and processes in order to ensure their correct and efficient operation. Aiming at supporting organizations in such a challenging task, the Information Technology Infrastructure Library (ITIL) [1], an important reference in this context, recommends, among other processes, change management. This process helps to efficiently handle the necessary changes in hardware and software within organizations. The scope of change management ranges from the change specification, by an initiator in a high level of abstraction, to the generation of detailed plans – those that comprehend low level activities such as modification of network device settings and C. Bartolini and L.P. Gaspary (Eds.): DSOM 2009, LNCS 5841, pp. 138–151, 2009. © IFIP International Federation for Information Processing 2009

CHANGEADVISOR

139

deployment of new services. These plans, once executed, tend to accomplish the changes requested by an initiator in the managed infrastructure. One of the main problems considering the generation of change plans is that, from the same high level specification, different detailed plans can be obtained. Although correct, they may not be necessarily aligned with the purposes defined for the IT infrastructure in the business level. The purposes can either have a technical nature, such as imposing constraints regarding the unavailability of provided services by the managed IT infrastructure, or define objectives for optimizing resource allocation during the change deployment (e.g., minimizing the number of human resources involved in an e-mail service maintenance procedure). When these purposes are not considered, the generated plans, if executed, may lead to results that violate policies predefined in the business level (e.g., e-mail service maintenance downtime must not exceed 10 hours monthly). We highlight three important reasons for aligning IT change plans with business purposes (also referred to as business objectives/constraints in this paper). First, there is the possibility of optimizing available resources that are usually limited and expensive. Second, the alignment has the potential of reducing costs such as the time spent to deploy changes. And finally, perhaps the most important reason, aligning change plans with distinct purposes allows the operator to understand the resulting trade-offs when one chooses a change plan among various possibilities. Despite the potential benefits, the alignment of change plans with business objectives/constraints has been neglected in investigations carried out in the area of IT service operations and management. The generation of executable [2] and optimized plans [3] are examples of issues that, alternatively, have been addressed recently. To tackle the aforementioned problem, in this paper we propose a solution for the alignment of IT infrastructure and service change plans with business objectives/constraints. In contrast to previous investigations conducted in the area, this paper focuses on the automated generation of change plans guided by business objectives/constraints. The proposed solution is experimentally evaluated through CHANGEADVISOR, a prototypical implementation of a decision support system that provides means for the understanding of trade-offs among alternative change designs. During the evaluation, which was performed considering scenarios based on real cases, we analyzed correction and completeness of generated plans, as well as their alignment with business purposes. The reminder of this paper is organized as follows. In Section 2 we review some key investigations in the area of IT change management. In Section 3 a conceptual solution for the generation of change plans aligned with business purposes is presented. We detail, in Section 4, the CHANGEADVISOR system and present, in Section 5, the results achieved using the system. Finally, we conclude the paper conveying final considerations and perspectives for future work in Section 6.

2 Related Work The IT service management area has grabbed the attention of the scientific community in recent years. Several aspects, such as models [4], automation [5], and alignment with business purposes [6] have been explored. Specifically analyzing the subarea of

140

R.C. Lunardi et al.

change management, it is possible to observe similar activity. For example, important investigations about scheduling [7] and rollback [8] can be cited. Nevertheless, the topic of IT change planning, which is the focus of this paper, has not been sufficiently explored, as underscored in the following two paragraphs. Keller et al. [3] have proposed CHAMPS, a system for the automated generation of change plans, which explores a high degree of parallelism in task execution. Although it has proved to be able to improve task scheduling considering available hardware resources, neither the activity association to human resources nor the objectives/constraints (specified by the change initiator) have been taken into account. In a recent paper, Cordeiro et al. [2] have proposed a solution to formalize, preserve, and reuse knowledge acquired with frequent IT changes. The solution, which is based on change templates, allows recurrent activities (e.g., service change or implementation) to be specified and (recurrently) reused. Although this solution comprises an algorithm to generate detailed plans from high level specifications, aligning generated plans with business purposes is out of its scope. In a subsequent work [9], the mentioned algorithm has been improved to consider technical constraints imposed by the managed environment (e.g., availability of resources such as disk and memory space). However, business purposes have not been considered. Other investigations, although not designed to deal with change management, have explored alignment with the business level. One of them is Keller’s [10], which introduced the concept of electronic contracts. In that research, the authors have proposed four kinds of contracts: RFCs, Deployment Descriptors, Policies and Best Practices, and Service Level Agreements (SLAs). Such contracts have different purposes, formats, and level of details. They are used for objective/constraint specification and are of great importance for the change deployment process. Summing up, although change management has been a focus of recent investigations, they have not considered alignment of changes with specific purposes in the business level. It means that the planning and subsequent change deployment may potentially be conducted in an undesired direction, i.e., may not be aligned with purposes demanded by the change initiator. In order to deal with this problem, the following sections present not only a conceptual solution, but also a system developed to support change alignment with business purposes and its evaluation.

3 Conceptual Solution Aiming to support the generation of change plans aligned with business purposes, two new key components have been introduced to the conceptual solution for change design and planning proposed in a previous work [9]. The new components – highlighted in gray – materialize the mechanism for the planning of changes guided by business purposes and fit adequately into the previously proposed solution, without significant modifications. Fig. 1 presents an overview of the extended solution, focusing on its main components, involved actors, and their interactions. The initiator starts a change process interacting with the component Change Designer (flow 1 in Fig. 1) to specify a Request for Change (RFC). RFCs describe what changes must be done in the managed infrastructure, the Configuration Items (CIs) primarily affected (network devices, computers, services, applications, etc.) and the

CHANGEADVISOR

141

business objectives to be met. However, the RFC does not specify details on how the change should be executed; such details have to be specified in the following step by the design of a preliminary plan, supported by the component Change Designer (flow 2). This plan consists in a workflow of actions that describe, in a high level of abstraction, how the requested change should be performed in the IT infrastructure. The generation of a detailed change plan based on the preliminary specification is then performed, without human intervention, by the component Change Refiner (3). The automated processing, carried out by a refinement algorithm, is based on both (i) information about software packages, available in the Definitive Media Library (DML) (4), and (ii) information about the target IT infrastructure, found in the Configuration Management System (CMS) (5). After the change refinement, the resulting plan is forwarded to the component Change Aligner (6), which plays a central role in the alignment of the plan with the business objectives/constraints expressed in the RFC. In order to guide this alignment, the Change Aligner retrieves from the CMS (7) information about costs and skills of available human resources (e.g., an operator who is an expert in e-mail service and whose work hour costs 40 monetary units). It also gathers information about the capabilities of the hardware assets affected by the plan (e.g., computational power in the case of workstations and servers). It is important to highlight that the process of change refinement and alignment is reiterated (8) during a period of time predetermined by the operator, aiming to generate distinct detailed plans aligned to the same business purposes determined in the RFC. This is due to the fact that the problem is NP-hard, and the algorithms employed to solve this problem are heuristic-based, i.e., the techniques used take advantage of information on past experiences in order to obtain results closer to the optimal (best) solution.

Change Initiator

Operator (2)

(11)

(9)

(1) Change Planner Change Designer

(3)

Change Refiner (4)

(6) (8) (5)

Definitive Media Library

Change Aligner (7)

(10)

Deployment System (13)

CIs

(12)

Configuration Mgmt. System

Fig. 1. Conceptual architecture of the proposed solution

In a later moment (9), the generated plans may be modified by the operator as to reflect his/her needs, and evaluated in terms of trade-offs between the various options. Among the criteria that may guide such an analysis, we cite: expected time and/or financial costs to perform the changes and human resources required. Based on this analysis, the operator will be able to select an appropriate plan to be deployed. Finally, in the last step of the change process, the operator invokes the deployment of the selected plan (10), interacting with the component Deployment System (11).

142

R.C. Lunardi et al.

This component may consume software packages available in the Definitive Media Library (12) in order to be able to execute some of the activities described in the change plan. After deploying this plan, the Deployment System updates the CMS, adding information about modifications in each of the affected CIs (13). It is important to mention that the conceptual solution depicted in Fig. 1 is targeted at the design, planning, and deployment of changes. Therefore, other phases that comprise the traditional change management process – such as the evaluation of the requested changes by the Change Advisory Board (CAB) and the schedule of changes into maintenance windows – are envisaged as directions and trends for long-term, future investigation in this area. Having presented an overview of our solution, in the following subsections we describe (i) the generation of detailed plans based on preliminary specifications and (ii) the process of aligning plans with purposes determined in the business level. 3.1 Change Refiner The component Change Refiner is fundamental in the process of generating alternative detailed change plans based on a preliminary specification. The core of this component is a refinement algorithm, inspired in a previous work [9]. As previously mentioned, the generation of detailed change plans requires not only information about the CIs (e.g., hardware, software, and people) that compose the IT infrastructure and their relations, but also about the software packages available for the change deployment. Aiming to represent this information in the CMS and in the DML, respectively, a subgroup of the Common Information Model (CIM) is used in this paper [11]. Fig. 2 presents a partial view of the model. Relationships such as associations, compositions, and aggregations, most of them omitted in the figure for better intelligibility, express dependencies among the elements of the infrastructure. MetricForME MetricDef ForME

* *

BaseMetric Definition 1 MetricInstance * BaseMetric * Value *

Managed Element

Organizational Entity

*

Managed SystemElement

User Entity

RoleLimited ToTarget * Role

Check

*

Logical Element

DiskSpace Check SwElement VersionCheck 0..1 1 Alternate SwDependency

SwChecks 1

EnabledLogical Element

Software Element

Computer System

Operating System

Execute Program

Service

Reboot Action

Action *

1

Action Sequence

Memory Check

ModifySetting Action

1

SwActions System

Setting Check

0..1

Human Action

Fig. 2. Information model used to represent the IT Infrastructure and the DML

The model also incorporates the classes Check and Action, which denote necessary information for computing dependencies refining the activities of a preliminary plan. An instance of the class Check defines a condition to be met or a characteristic required for the associated software to evolve into a new state (deployable, installable, executable or running).

CHANGEADVISOR

143

Fig. 3(a) illustrates instances of Check associated to the software Squirrelmail. The first two indicate that the software packages Exim4 and Apache2 must be installed before the Squirrelmail installation. The last two express that Squirrelmail requires 1MB of memory for installation and 3MB of disk space for execution. Each instance of the class Action, in turn, represents a mandatory activity to change the state of the associated software (e.g., from installable to executable, i.e., an installation process). Using once more the example of Squirrelmail, two activities associated with its installation process are illustrated in 3(a): Install Squirrelmail, responsible for the copies of Squirrelmail binary files; and Configure Squirrelmail, which performs the configurations necessary for the webmail to operate properly. MetricDefForMe

Squirrelmail: SwElement SwChecks SwActions

SwElement: Check – name: Exim4

Install Squirrelmail: Value:MetValue Action MetricForMe – MetricValue: 1

SwElement: Check – name: Apache2

ComputingPower: MetDefinition

SwChecks

Memory: Check – amount: 1 MB

MetricDefForMe ComputingPower:

MetDefinition

SwActions

– amount: 1 MB

DiskSpace: Check – amount: 6 MB

MetricInstance

Install Exim4: Action

Value:MetValue

Config Exim4: Action

MetricForMe Value:MetValue – MetricValue: 5 MetricInstance

MetricForMe – MetricValue: 1 ActionSequence

ActionSequence

Config Squirrelmail: MetricForMe Value:MetValue Action – MetricValue: 3

Memory: Check

Exim4: SwElement

MetricInstance

MetricInstance

DiskSpace: Check – amount: 3 MB

MetricDefForMe

ManPower: MetDefinition

MetricDefForMe

(a) Apache2: SwElement SwChecks

Memory: Check – amount: 1 MB

DiskSpace: Check – amount: 3 MB

(b) MetDefinition MetricInstance

SwActions

Install Apache2: MetricForMe Value:MetValue Action – MetricValue: 2

A: UserEntity – Name: A

MetricInstance MetricDefForMe

ManPower: MetDefinition

RoleLimited ToTarget

ManPower/h: MetDefinition

MetricInstance

Value:MetValue

MetricForMe

- MetricValue: 120

ActionSequence

Config Apache2: Value:MetValue Action MetricForMe – MetricValue: 1

MetricDefForMe

Expert: Role

MetricDefForMe ComputingPower:

(c)

ManPower: MetDefinition

MetricDefForMe

Junior: Role B: UserEntity – Name: B

C: UserEntity

ManPower/h: MetDefinition

MetricInstance MetricForMe

RoleLimited ToTarget

Value:MetValue – MetricValue: 90

– Name: C

(d)

Fig. 3. Software Elements present in DML (a,b,c) and roles in the IT infrastructure (d)

The algorithm for change refinement, also known as change_refinement, receives as input the change specification to be refined, along with data about the current state of the managed infrastructure (from CMS) and the software packages from the DML. For each activity composing the preliminary plan, change_refinement computes the set of activities that should be executed before, so as to satisfy the requirements (e.g., installation of software dependencies) for the activity to be executable. To illustrate the refinement process, consider the RFC Migrate E-mail Server, presented in Fig. 4 (background in grey). It consists in the migration of the e-mail service from server X to server Y, and aims at minimizing the deployment time estimated in, at most, 120 minutes (in a way not to compromise the normal operations of the organization). The preliminary plan associated (background in white) is composed of six high level activities that range from the installation and configuration of a new server to the migration of data from the “old” (X) to the “new” server (Y). Focusing on the activity Install Squirrelmail (illustrated in the preliminary plan), change_refinement identifies – through the Checks presented in DML (Fig. 3(a)) – that the execution of this activity depends on the previous installation of software

144

R.C. Lunardi et al.

RFC Name: Migrate E-mail Server has operation Operation Objective: ExecutionTime Name: Migrate E-mail ThresholdValue: 120 minutes has change plan

from

X: ComputerSystem

to

Y: ComputerSystem

Activity Name: Install Web Server Activity Name: Build New Server

Activity Name: Add Users

Activity Name: Migrate Webmail

Activity Name: Copy Users Data Activity Name: Install E-mail Server

Fig. 4. Example of RFC and preliminary change plan for the migration of an e-mail server

packages Apache2 (Web server) and Exim4 (e-mail server). Consequently, two new activities are generated to satisfy the demands of Install Squirrelmail: Install Apache2 (a) and Install Exim4 (b). This process goes on recursively until all the requirements of the activities of the preliminary change plan are met. At the end of this process, when a detailed plan is obtained (Fig. 6), change_refinement stores the “current state” (refinement paths traversed to generate this plan). This step is important because, after the alignment (seen in detail in the Subsection 3.2), it will be possible to generate distinct, detailed plans, from the same initial specification, using alternative refinement “paths”. 3.2 Change Aligner The component Change Aligner, a key element for the alignment of detailed plans with business purposes, is grounded in three building blocks: (i) formalization of business objectives/constraints, (ii) information models to represent cost/capacity metrics, and (iii) algorithm for change alignment. These are described next. 3.2.1 Business Objectives/Constraints and Cost/Capability Metrics To make it possible for change plans generated by the component Change Refiner to be aligned with business purposes, these must be specified in the RFC. Two business purpose categories are defined in this first iteration to solve the problem: Execution Time and Execution Cost to perform the change. These objectives were indicated as the most relevant ones by a group of network/system managers (working in Brazilian IT companies) interviewed. The specification of an objective/constraint needs to be accompanied by a Threshold Value representing the maximum time or cost tolerated. As shown in Fig. 4, the RFC Migrate E-mail Server specifies that the change, when planned, should consider at most 120 minutes for deployment. Cost and capability metrics are also essential for the alignment to be performed. Whereas the former are used to describe the effort demanded in the execution of activities related to IT service operations and management, the latter allow expressing expectation on the performance of humans and computers to execute them. The metrics employed in this paper, described next, have already been adopted and consolidated in areas such as Distributed Systems [12], Software Engineering [13], and Economy [14]. Concerning information modeling, the metrics are specified, and their values instantiated, in the IT infrastructure model (Fig. 2) by means of the classes Base Metric Definition and Base Metric Value.

CHANGEADVISOR

145

The cost metrics are subdivided in Man Power (MP) and Computing Power (CP). MP is used to quantify human effort required for executing an activity. CP, in turn, denotes computational effort required for executing an automated activity, i.e., with no (or minimal) human intervention. Value instances for these metrics are used to express the costs of activities associated with software elements registered in the DML. In Fig. 3, it is possible to observe, as an illustration, that activities like Install Squirrelmail (a), Install Exim4 (b), and Install Apache2 (c) have an associated execution cost, in CP, of 1, 1, and 2 units, respectively. Additionally, the activities for the configuration of these services, namely Config Squirrelmail (a), Config Exim4 (b), and Config Apache 2 (c), have an associated cost, in MP, of 3, 5, and 1 units. Similarly to what happens to cost metrics, the capability metrics are also subdivided in two: Man Power/hour (MP/H) and Computing Power/hour (CP/H). They represent the amount of work required from humans and computers to perform the execution of activities. In the case of MP/H, instance values are associated with roles played by the board of human resources available in the IT department of the organization. In the example illustrated in Fig. 3 (d), two roles are modelled, Expert and Junior, with MP/H equal to 120 and 90 units, respectively. Human resources are then associated to these roles, according to the functions they are able to perform in the department. MP/H instance values, conversely, are attributed to each of the computers present in the IT infrastructure, reflecting its expected performance. We highlight that the investigation about ways of valuing costs and capabilities, although fundamental for the proposed solution, are not the scope of this paper. To fill in this gap, we assume the employment of one of the approaches proposed in the literature (some of them have been compiled by Jorgensen [15]). More specifically, in this paper, we adopted an empirical method, which consists of valuing simpler activities – not only automated but also of human-based execution – assigning to them a value of 1 CP and 1 MP, respectively. Having this as a basis, costs of other activities are estimated. The estimation of computing resource capabilities (CP/H) and human ones (MP/H), in turn, is performed by analysing historic data. 3.2.2 Alignment of Change Plans with Business Objectives/Constraints The building blocks previously presented form the basis upon which the algorithm for aligning change plans with business purposes is built. The algorithm, shown in Fig. 5, is denoted by change_alignment (C, DML, IT, H), where C is the refined plan generated by change_refinement; DML is the repository of that contains information about software packages available; IT is the repository of information about the managed infrastructure; and H is the set of human resources available for conducting changes. As a first step of the alignment process, an ‘empty’ change plan is created (line 1 in Fig. 5). After that, the activities contained in C are copied to the set D (line 2), and the function recursive_alignment is invoked (4). The alignment process, as described in recursive_alignment, starts by verifying if both D (the set of activities which have not been processed) is empty and plan C’ received satisfies the business objectives determined in the RFC (line 7). If so, plan C’ is returned as a solution. Otherwise, the following steps are executed. First, an activity ai is extracted from set D for processing (line 9). Second, the algorithm verifies if ai requires a human to be executed (line 10). In case it does, the algorithm tries to allocate the most capable human h (available in H, the group of human resources), who has not

146

R.C. Lunardi et al.

been allocated in any other parallel activity to ai (lines 11-18). Human resources are allocated in a descending order in relation to the role they are associated with. Analyzing the roles Junior and Expert illustrated in Fig. 3(d), the most capable humans belong to Expert’s. It is important to mention that, for the alignment computation, the allocation of humans is conducted in a “non-nominal” way, i.e., there is the specification that a human playing role x should execute a set of activities. The effective (or nominal) allocation of human resources to the activities is postponed to the change scheduling stage, which is out of the scope of this paper and has been investigated by several related work [3, 7]. 01 change_alignment (C, DML, IT, H) 02 C' ← empty change plan 03 D ← set of activities from C 04 recursive_alignment (C, C', D, DML, IT, H) 05 06 recursive_alignment (C, C', D, DML, IT, H) 07 if D is empty, cost of deploying C' comply with objective threshold, given DML, IT, then 08 return C' as solution 09 extract activity ai from D 10 if ai is a human activity then 11 for all h in H do 12 for all aj in C do 13 allocable ← true 14 if ai is parallel to aj in C and h is allocated to aj then 15 allocable ← false 16 if allocable is true then 17 associate human h to ai 18 add activity ai to change plan C' 19 if cost of deploying C' comply with objective threshold, given DML, IT, then 20 recursive_alignment (C, C', D, DML, IT, H) 21 else 22 add activity ai to change plan C' 23 if cost of deploying C' comply with objective threshold, given DML, IT, then 24 recursive_alignment (C, C', D, DML, IT, H) 25 return failure

Fig. 5. Algorithm for the alignment of change plans with business objectives/constraints

Once a human is allocated to ai, the algorithm verifies the cost of executing plan C’ on the managed infrastructure, considering such allocation (line 19). If the cost does not exceed Threshold Value (specified in the RFC), the alignment process continues recursively (line 20), in order to process the remaining activities in D. The semantics of cost (and threshold) depends on the RFC objective/constraint. For example, if Execution Time is chosen, then cost and threshold are processed considering time units. Alternatively, the activity ai extracted from set D may not require human intervention (line 21). In this case, it is directly added to the plan that will be returned as a solution (line 22). The processing continues recursively (line 24), in case the execution cost of C’ is aligned with the defined objectives (line 23). If none of the humans available can be allocated to ai or the cost for executing C’ (added to ai) exceeds Threshold Value, the alignment process fails (line 25). Consequently, the allocation of human resources done up to this moment are recursively rolled back, and new alternatives for the alignment are explored. In the worst case, recursive_alignment is not able to produce a plan aligned with the business purpose specified in the RFC, in which case a detailed feedback is returned to the operator.

CHANGEADVISOR

Activity Install OS Debian Linux

Activity Get apache2utils from repository

Activity Invoke apache2-utils installer

Activity Configure apache2-utils

Activity Get apache2common from Repository

Activity Invoke apache2common inst.

Activity Configure apache2common

Activity Add all users by script

Activity Get apache2 from repository

Activity Invoke apache2 installer

Activity Configure apache2

Activity Copy /var/www from labcom

Activity Configure DNS

Activity Restart DNS

Activity Copy All users data by script

Activity Get PHP5common from repository

Activity Invoke PHP5common installer

Activity Configure PHP5-common

Activity Get libapache mod-php5 from repository

Activity Invoke libapache modphp5 Installer

Activity Configure libapache modphp5

Activity Get PHP5 from repository

Activity Invoke PHP5 Installer

Activity Configure PHP5

Activity Get exim4config from repository

Activity Invoke exim4config installer

Activity Configure exim4-config

Activity Get exim4base from repository

Activity Invoke exim4base installer

Activity Configure exim4-base common

Activity Get exim4daemon-light from repository

Activity Invoke exim4daemon-light installer

Activity Configure exim4daemon-light

Activity Get exim4 from repository

Activity Invoke exim4 installer

Activity Configure exim4

Activity Copy /var/mail from labcom

Activity Configure DNS

Activity Restart DNS

Activity Get squirrellocales from repository

Activity Invoke squirrellocales installer

Activity Configure squirrel-locales

Activity Get squirrelmail from repository

Activity Invoke squirrelmail installer

Activity Copy Squirrelmail Settings

Activity Configure DNS

Activity Restart DNS

Unrefined Activity

Automated Activity

Manual Activity

Expert Webserver Operator Role

147

Junior Webserver Operator Role

Computer Operator Role

Fig. 6. Detailed change plan for the RFC Migrate E-mail Server

To illustrate the alignment process, consider the change plan presented in Fig. 6, generated for the RFC Migrate E-mail Server (Fig. 4). The different human roles allocated for the plan execution are represented using hachured areas around the activities, which have a grey background when human intervention is required. In this example, there is only one human associated with each role defined in H, being Expert Webserver Operator, Junior Webserver Operator, and Computer Operator the roles from where humans have been chosen. The presented plan meets the Execution Time objective, according to what was specified in the RFC. Note that the activities related to the installation of Exim4 (e-mail system) and that require human intervention have a higher cost to be executed. Therefore, they are associated with the most capable available human (belonging to role Expert Webserver Operator). In the same way, the allocation of humans for other activities in the plan is done considering the existent parallelisms, as well as the associated costs.

4 Implementation The solution to support alignment of change plans with business purposes presented in Section 3 has been implemented as a prototype of a decision support system called CHANGEADVISOR. This system is a key part of CHANGELEDGE [2], a framework for change management focused on the reuse of knowledge and automation. In this section, we present some of the main aspects of the implemented system, as well as its interfaces with the CHANGELEDGE components.

148

R.C. Lunardi et al.

CHANGEADVISOR, implemented using the Java programming language, performs the functionalities of the components Change Refiner and Change Aligner (grey area of Fig. 1). The change specifications consumed by the system are coded using XML and processed through the Streaming API for XML (StAX). From the received specifications, the system generates detailed change plans (executable workflows), which are also coded in XML and respect the Business Execution Language (BPEL) [16]. The choice for BPEL is due to the widespread use of the standard and to its suitability for coordinating distributed activities in IT infrastructures. In regard to data handling and persistence (e.g., about the managed infrastructure and the software packages available), both the Hibernate object-relational mapper and the MySQL database management system have been employed. The communication between CHANGELEDGE components and the CHANGEADVISOR system is done through well-defined interfaces. Among the existing interfaces, the ones that “connect” the system with the components Change Designer and Deployment System are highlighted. In the former, Change Designer generates XML documents, adopting as a pattern the same XML schema as CHANGEADVISOR. In the latter, the BPEL documents generated by CHANGEADVISOR are consumed by the orchestrator ActiveBPEL, which implements the Deployment System functionalities. Please refer to [8] for additional information concerning the operation of the Deployment System.

5 Experimental Evaluation In order to evaluate the technical feasibility of IT change planning aligned with business objectives/constraints, we have carried out several experiments using CHANGEADVISOR. RFCs with different objectives/constraints have been submitted, considering the same IT infrastructure as a basis. For space limitations, the analysis is focused on two of the changes. Table 1. Number of humans, cost, and capacity per role Role

Number of Available Operators

Allocation Cost ($ / hour)

Expert Webserver Operator

1

40

ManPower / Hour 2

Junior Webserver Operator

1

25

1.5

Computer Operator

3

12

1

Table 2. RFCs used in the experimental evaluation Scenario

RFC

Goal

Objective/Constraint

Threshold Value

1

1

Migrate Webmail from server A to C

Reduce Execution Time

120

2

1

Migrate Webmail from server A to C

Reduce Execution Cost

15

3

1

Migrate Webmail from server A to C

None

-

4

2

Install Web Application in Server B

Reduce Execution Time

120

5

2

Install Web Application in Server B

Reduce Execution Cost

15

6

2

Install Web Application in Server B

None

-

CHANGEADVISOR

149

The IT infrastructure employed in this evaluation is composed by 3 servers – A, B and C – out of which only A has Debian GNU/Linux installed. There are also three roles available, from which humans resources may be chosen to assist change execution. Table 1 shows the number of humans playing each role, the capacity, and the allocation cost per hour (per role). Table 2 describes the scenarios employed in the evaluation, highlighting the goals of the RFCs, as well as their respective objectives/constraints. A partial view of the plans generated for RFC 1, in scenarios 1 and 2, is presented in Figs. 6 and 7, respectively. Due to space restrictions, the refinement of Install Debian GNU/Linux (Fig. 6) and Install Windows 2003 Server (Fig. 7) is suppressed in both figures, as well as decision structures. The activities present in the illustrated plans are consistently connected and ordered, respecting the dependency information specified in the DML. Note, for example, that the installation of apache2 in Debian GNU/Linux is correctly preceded by the installation of its basic libraries.

Activity Install Microsoft Windows 2003

Activity Add all users by script

Activity Copy all users data by script

Activity Open windows component box

Activity Select ASP.NET

Activity Select IIS

Activity Select IIS Manager

Activity Select NNTP Service

Activity Select SMTP Service

Activity Select World Wide Web Service

Activity Invoke windows comp. Installer

Activity Enable IIS

Activity Enable ASP.NET

Activity Reboot Computer

Activity Copy /var/www from labcom

Activity Configure DNS

Activity Restart DNS

Activity Insert Exchange Server CD

Activity Configure Exchange Server 2003

Activity Invoke Exchange Server 2003

Activity Get Exchange Server 2003 SP2

Activity Invoke Exchange Server 2003

Activity Copy /var/mail from labcom

Activity Convert e-mail files

Activity Configure DNS

Activity Restart DNS

Unrefined Activity

Automated Activity

Manual Activity

Expert Webserver Operator Role

Fig. 7. Alternative change plan for the RFC Migrate E-mail Server

One can also observe in Figs. 6 and 7 that different human roles are required for the execution of different “niches” of manual activities, so as to meet the specified business objective/constraint. For example, the change plan generated for scenario 1 requires three humans, each playing a different role (one Expert Webserver Operator, one Junior Webserver Operator, and one Computer Operator) for the determined objective to be met. Conversely, the plan for scenario 2 requires only one operator (Expert Webserver Operator) to meet the determined objective. Table 3. Estimated execution time and human allocation cost for the generated detailed plans Scenario

RFC

Activities

Estimated Execution Time (min)

Human Allocation Cost ($)

1

1

69

118.75

34.21

2

1

38

156.12

7.75

3

1

68

157.08

45.98

4

2

52

75.5

18.73

5

2

39

211.31

10.42

6

2

49

250.75

28.43

150

R.C. Lunardi et al.

Table 3 presents costs associated to the execution of the generated change plans for each of the evaluated scenarios. Observe that the resulting detailed plans exhibit satisfactory results in relation to the determined business purpose. For instance, RFC 1 could be accomplished by two distinct plans: one that has a shorter execution time (118 min), although having a higher implementation cost ($34.21), or another one which has a reasonable cost ($7.75), in spite of demanding a longer execution time (156 min). If analyzed individually, these cost differences may be insignificant. However, assuming that many changes may be required on a daily basis within an organization, such an alignment has the potential to reduce operational costs significantly. Table 3 also enumerates, for the sake of comparison, change plans not aligned with objectives/constraints. Although correct, such plans have higher costs if compared to the plans that follow any alignment. For example, the plan generated for scenario 3 has an estimated cost of $45.98 with human allocation, an amount 493% higher than the cost of the aligned plan. This is explained by the different possible refinements for a same preliminary plan. If objectives/constraints are not taken into account, the exploration space gets wider, possibly leading to plans that violate business objectives/constraints. In regard to performance, CHANGEADVISOR has required from a few hundreds of milliseconds to a few seconds to generate each aligned change plan. This is a negligible processing cost, especially if compared to the time humans would require to do the same job.

6 Conclusions and Future Work Change planning is a fundamental element for the operation and management of Information Technology infrastructures and services. Existing automated solutions, however, do not take into consideration the generation of change plans aligned with objectives determined in the business level. Consequently, the planning and execution of changes will present little (or no) relation with the objectives/constraints determined by the change initiator. In order to tackle this problem, in this paper we have proposed CHANGEADVISOR, a solution for generation of change plans, which are consistent with objectives/constraints determined in the business level. The results obtained, although not exhaustive, confirm the potential benefits of aligning change plans with business purposes. As previously mentioned, several detailed plans have been obtained from a same RFC, each of them satisfying specific execution time and cost requirements. One may also observe that the generated plans were sensibly different in form (comprising activities) as well as in associated costs (time and financial costs). Finally, the system has performed satisfactorily with respect to the time spent in the generation of the plans (kept in the magnitude of seconds). This time is certainly shorter than the time that would be spent by an experienced operator to manually design the plans using a workflow editor. As future work, we intend to: (i) extend the scope of the alignment process so as to support two or more simultaneous objectives/constraints; (ii) investigate a richer set of objectives/constraints; and (iii) explore different heuristics (e.g., for human resource allocation) to be used by the alignment algorithm.

CHANGEADVISOR

151

References 1. Information Technology Infrastructure Library. ITIL V3. Office of Government Commerce (OGC) (2008), http://www.itil-officialsite.com 2. Cordeiro, W., Machado, G., Daitx, F., et al.: A Template-based Solution to Support Knowledge Reuse in IT Change Design. In: IFIP/IEEE Network Operations and Management Symposium (NOMS 2008), pp. 355–362 (2008) 3. Keller, A., Hellerstein, J.L., Wolf, J.L., et al.: The CHAMPS System: Change Management with Planning and Scheduling. In: IEEE/IFIP Network Operations and Management Symposium (NOMS 2004), pp. 395–408 (2004) 4. Rodosek, G.D.: A Generic Model for IT Services and Service Management. In: IFIP/IEEE International Integrated Network Management (IM 2003), pp. 171–184 (2003) 5. Brown, A.B., Keller, A.: A Best Practice Approach for Automating IT Management Processes. In: IFIP/IEEE Network Operations and Management Symposium (NOMS 2006), pp. 33–44 (2006) 6. Moura, A., Sauvé, J., Bartolini, C.: Business-driven IT Management - Upping the Ante of IT: Exploring the Linkage between IT and Business to Improve both IT and Business Results. IEEE Communications Magazine 46(10), 148–153 (2008) 7. Trastour, D., Rahmouni, M., Bartolini, C.: Activity-Based Scheduling of IT Changes. In: Bandara, A.K., Burgess, M. (eds.) AIMS 2007. LNCS, vol. 4543, pp. 73–84. Springer, Heidelberg (2007) 8. Machado, G., Daitx, F., Cordeiro, W., et al.: Enabling Rollback Support in IT Change Management Systems. In: IFIP/IEEE Network Operations and Management Sympo-sium (NOMS 2008), pp. 347–354 (2008) 9. Cordeiro, W., Machado, G., Andreis, F., et al.: A Runtime Constraint-Aware Solution for Automated Refinement of IT Change Plans. In: De Turck, F., Kellerer, W., Kormentzas, G. (eds.) DSOM 2008. LNCS, vol. 5273, pp. 69–82. Springer, Heidelberg (2008) 10. Keller, A.: Automating the Change Management Process with Electronic Contracts. In: IEEE International Conference on E-Commerce Technology Workshops (CECW 2005), pp. 99–108 (2005) 11. Distributed Management Task Force (DMTF): Common Information Model, http://www.dmtf.org/standards/cim 12. Shepherd, M.: Special Feature Distributed Computing Power: a Key to Productivity. IEEE Computer 10(11), 66–74 (1977) 13. Pillai, K., Sukumaran Nair, V.S.: A Model for Software Development Effort and Cost Estimation. IEEE Transactions on Software Engineering 23(8), 485–497 (1997) 14. Banyahia, H.: Costs and Productivity Estimation in Computer Engineering Economics. Engineering Economist 41(3), 229–241 (1996) 15. Jorgensen, M., Shepperd, M.: A Systematic Review of Software Development Cost Estimation Studies. IEEE Transactions on Software Engineering 33(1), 33–53 (2007) 16. Organization for the Advancement of Structured Information Standards. OASIS: Business Process Execution Language, version 2.0, http://docs.oasis-open.org/wsbpel/2.0

Towards Robust IT Service Portfolio Management David Trastour and Athena Christodoulou HP Labs, Bristol, UK [email protected], [email protected]

Abstract. ITIL recommends implementing Service Portfolio Management as a way to actively manage IT investments, with transparency in its operations and spending. Selecting which and when projects should be executed is an important but complex task. It is all the more complex if one includes operational costs, which represent a significant component of the IT budget. We investigate techniques to provide decision support in selecting IT investments. We provide a model to link IT investment selection to business value and a method to generate valid portfolios and to guide an IT executive the ‘best’ portfolio. Finally, we evaluate our method with a case study and report on the feasibility of the method. Keywords: IT Portfolio Management, Business Service Management, Robust Optimisation.

1 Introduction While it is true that IT enables and enhances an increasing number of goods and services, too often IT organisations struggle to demonstrate the business value of IT since interactions between business and IT stakeholders are often centred on technical capabilities. The IT Information Library (ITIL) [1] makes a clear distinction between Business Services that generate value and take part in business processes and IT Assets that underpin other services but have no business context. For instance, a credit rating service delivers value to a bank and is a business service; an infrastructure service, such as a shared database service, is an IT asset since it does not provide any direct business value. Several business services may however depend on the shared database service. Up to now, IT organisations have been focused on operating IT assets (networks, servers, and storage) and have given little attention to business value. IT executives are increasingly under pressure to justify the costs of IT and need to transform IT assets into business services, with clearly identified costs and business benefits [2]. ITIL recommends implementing Service Portfolio Management [3] as a way to actively manage IT investments, with transparency in its operations and spending. IT investments can be categorised into two categories. Operational costs are required to manage and maintain IT assets that are listed in the Service Catalogue, for instance to deal with incidents, problems and changes. IT projects are carried out for the development of new services and retiring services that are no longer economically or technically viable. C. Bartolini and L.P. Gaspary (Eds.): DSOM 2009, LNCS 5841, pp. 152–163, 2009. © IFIP International Federation for Information Processing 2009

Towards Robust IT Service Portfolio Management

153

When doing their yearly planning, IT executives consolidate the demand for IT resources, both for project and operations activities, that emanates from the various business units and from IT, and select how to best allocate budget and resources. It is not unusual to have several hundreds of such demands in large IT organisations. Because of the considerable amounts of resources that are at stake, planning IT investments is an important but complex task. We have identified four reasons that make this decision complex. First, complex inter-dependencies exist between IT assets, and between business services IT assets, making it difficult to link IT investment to strategic business planning. Secondly, because the planning process can happen between one to two years ahead of execution, there is a high level of uncertainty in the input data. For instance, costs, workloads and durations of new IT projects can only be estimated and may be widely variable. Third, the selection of IT investments is subject to multiple constraints, resource constraints, budget constraint, time constraints or interdependency constraints. Finally, there are many criteria to take into account in evaluating a portfolio: a CIO may want to provide the most valuable services while keeping costs low and maintaining high levels of employee satisfaction. In a typical scenario, such objectives are conflicting, so a trade-off must be found. In this paper, we discuss the requirements of IT service portfolio selection and concentrate on ways to link IT investment selection to business value. We then implement a multi-objective selection method to generate feasible portfolios and guide IT executives in finding the portfolio that best aligns to their preferences. The method addresses issues of information incompleteness and uncertainty. This paper is organised as follows. Section 2 reviews related work, and section 3 presents the decision problem and the related information model. Section 4 shows the formulation of the optimisation problem, and how the method can be applied interactively to provide decision support. The method is illustrated through a simple example in section 5. Finally, we conclude and present our future work intentions.

2 Related Work This work contributes to the research domain of Business-Driven IT Management (BDIM). BDIM has been defined as “the application of a set of models, practices, techniques and tools to map and to quantitatively evaluate interdependencies between business performance and IT solutions – and using the quantified evaluation – to improve the IT solutions’ quality of service and related business results” [4]. The BDIM paradigm has already been applied to several IT domains: in change management, IT changes can be evaluated and scheduled in order to minimise financial losses [5][6], in capacity management, systems configuration can be optimised from a business objective perspective [7][8], in network management, network configurations must align with customer satisfaction and business profitability [9], or in incident management, performance of IT support organisations is evaluated and optimised to meet service levels [10]. The latter example is notable in that it does not address the technology dimension of IT but rather the aspects of people and processes. In line with these approaches, we build a linkage model between an IT decision − whether to select an IT investment or not − and business outcome.

154

D. Trastour and A. Christodoulou

This work also relates to a number of fields of Operations Research and Management Science. The IT investment decision problem is linked with financial investment planning [11][12], but there are significant differences in the problem formulations and in the solution methods. First, the decision variables for IT projects are binary (go/no-go decision) while they are continuous in financial planning (corresponding to an amount of security). Secondly, financial portfolio selection heavily relies on past performance to forecast future performance and understand the interdependencies between different securities. There is much less consistency in the IT domain since very few projects are repeated. Finally, while investing in a given financial asset affects the portfolio performance, it does not significantly influence the market performance of the asset. The amount of resources invested in an IT project can determine its outcome and the quality of delivered assets. R&D project selection and resource allocation have received considerable attention over the last three decades. Various models and algorithms have been developed to help a decision maker in selecting R&D projects from a pool of available projects namely: traditional scoring techniques [13], multi-attribute utility theory [14], comparative approaches such as q-sort [15] or analytical hierarchy procedure [16]. Multiple optimization models based on mathematical programming have also been proposed [17]. We refer the reader to [18] for a survey of this diverse set of techniques. Nevertheless, very few works deal with the requirements presented in the introduction: linking IT assets to business value, dealing with uncertainty, constraints and multiple conflicting objectives. A notable exception is the work of Stummer and Heidenberger [19] which apply multi-objective optimization techniques [20] to the problem of project selection. Combining different evaluation criteria, such as return-on-investment (ROI) or customer satisfaction, into one objective is a difficult task. A decision maker cannot easily assign target regions and relative weights for each criterion, a priori [21]. Rather, she needs to explore the solution space, and participate in the search process. More recently, in [22], the authors keep the principles found in [19] but extend the work by formalising the notion of dominance for project portfolio section and by presenting a scalable algorithm to compute nondominated portfolios. Considering that, in most firms, IT projects account for less than 30% of the total IT spend, and that the remaining 70% goes towards maintenance and ongoing operations [2], it is not sufficient to only focus on the selection of IT projects. In this paper, we provide a method for the selection of all IT investments, addressing simultaneously projects and asset operations.

3 Conceptual Solution for Service Portfolio Management We propose a conceptual solution for service portfolio management that takes into account all IT investments, both project activities and on-going maintenance activities. We first describe the information model that is used in the remainder of this paper, then propose a model that links the selection of an IT investment to business objectives, discuss the data uncertainty, and conclude with possible metrics to evaluate IT portfolios.

Towards Robust IT Service Portfolio Management

155

3.1 Information Model The proposed information model includes the main concepts necessary in service portfolio management. We chose to base the model on ITIL terminology and concepts whenever appropriate. Fig. 1 depicts this information model. Following the ITIL guideline, we make a distinction between IT services that provide business value (business services), and internal services whose purpose is to provide infrastructure and necessary underlying functionality with no direct business value (IT assets). A business process is composed of a number of business services and in this paper we only consider business services that are supported by IT. Business services are composed of one or several assets, and assets in turn can depend on other lower- level assets. Because the goal of this work is to guide the selection of IT investments, the intent is not to have a fine grained model of assets, as one would find in a configuration management database (CMDB). The configuration items present in the CMBD are elements of the infrastructure (e.g. servers and routers) and of the software (e.g. operating systems and applications). Here, an IT asset is a coarse-grain representation of an IT service that includes the people operating it (support staffs, possibly from different departments), the underlying technology (configuration items) and processes (IT service management processes such as incident management or change management). Metrics are used to measure the performance, availability, maintainability and regulatory compliance of business services. In this work, we are agnostic as to what the metrics are. They can represent service levels such as Mean Time To Restore (MTTR) or Mean Time Between Failures (MTBF) frequently used to measure the maintainability and availability of services, financial metrics such as cash flow or sales, or performance indicators of IT processes. The Control Objective for Business Information related Technology (COBIT [23]) specifies a large collection of such metrics and high-level performance indicators. We assume that executives have opinions about the objectives they are trying to achieve and have ideas about suitable target regions for each metric. For instance, executives may wish to increase revenue by 10% and improve customer satisfaction (measured by doubling the MTBF of the ecommerce site). An IT investment is an activity that consumes budget and resources. For each investment, a cost profile and a staffing profile are defined. The cost profile defines the cost of the project as a function of time. In the context of this paper, time is discrete and the unit is typically in months or quarters. The staffing profile gives an estimation of the required human resources as a function of time. Because IT resources are highly specialised, staffing profiles are organised per role (e.g. database administrator, level 1 PC support, or Java developer). IT investments are specialised into two categories, operations and projects. An operations investment ensures the smooth execution of an asset. Several operations investments can be proposed for each asset and only one will need to be selected. As will be detailed in the following sub-section, different operations investments, hence different levels of budgets and resources, will lead to different levels of business service metrics. A project aims at developing a new service, enhancing an existing service or decommissioning a service. Projects can have impact on several metrics. For instance, a project that deploys a new feature on a firm’s e-commerce web site may generate additional revenue and improve customer

156

D. Trastour and A. Christodoulou

Fig. 1. Information model for service portfolio management

churn rate. Projects may also reduce operating costs, and will have impacts on the cost and resource profiles of assets, with IT consolidation projects being a typical example. 3.2 Decision Problem In this sub-section, we introduce some notation and formalise the decision problem we are solving. Let be the number of business services and be the number of assets being considered. Let be the set of assets on which service depends, i.e. the transitive closure over the dependency relation. Let be the number of metrics for services. Finally, let be the number of projects and let be the number of operations investment associated with asset . Throughout this paper, we will use the indices for services, for assets, for operations investments, for projects, for metrics and for time period. We wish to select the set of investments over a number of time periods that will lead to the ‘best’ outcomes. To this end, we define the Boolean decision variables: •

1, , 1, , such that , , is selected for asset ; • 1, , 1, , such that , , , it is scheduled to start at time period .

,

1 if and only if operations investment 1 if and only if project

is selected and

A solution of the service portfolio selection problem is hence an assignment of the variables , and , . The set of feasible portfolios is defined by refining the set of all possible assignments and ensuring that the following types of constraints apply:

Towards Robust IT Service Portfolio Management

157

• Operations constraints: one and only one operations investment must be selected for each asset; • Budget constraints: for each time period, the total spend in projects and operations must not exceed the planned budget; • Resource constraints: for each time period, the total demand for resources must not exceed the planned resource supply; • Logical constraints: rigid interdependencies between projects and operations investment (e.g. follow-up projects and mutually exclusive projects); • Positioning constraints ensure that the portfolio is aligned with strategic requirements and that it well balanced between business unit, functional area, geography, or technology area; • Threshold constraints filter out investments that do not meet minimum acceptable levels (for instance the Net Present Value (NPV) must exceed a given threshold). These constraints are common in portfolio selection [22] and can be modelled as linear inequalities using the decision variables , and , .

4 Multi-criteria Service Portfolio Selection under Uncertainty The conceptual solution presented above does not define the criteria that qualify the ‘best’ portfolio. Moreover, we have not addressed our requirement to deal with the uncertainty in the input data. In this section, we address these issues: we first comment on aspects of uncertainty and risk, define criteria to evaluate service portfolios and propose a solution method. 4.1 Uncertainty and Risk The level and significance of uncertainty differs among specific assets and projects, with the demands for asset operation demands and outcomes being far more predictable than IT projects. Also, the level of uncertainty may not be the same for all projects. Clearly uncertainty will be in its highest when a project is at the proposal stage; however it will continue to exist through its execution and delivery. We also note that uncertainty comes in many forms in IT service portfolio management. Not only the human and monetary demands of projects and assets cannot be fully known in advance and their outcome is not certain, but unforeseen events may the expected outcomes. For instance technical difficulties may require additional development, quality issues may require additional testing, or supplier delays may postpone the completion of a project. Sophisticated models that take into account project and asset specific knowledge such as these unforeseen events and risks could be built to represent costs, resource demand and business impact. In this paper, we do not mandate what method is used to build these models. In practice, we believe that confidence intervals may be a simple enough approach that it may be accepted by practitioners, while at the same time capturing more information that a deterministic model. The shapes of the distributions could be either assumed (for instance Beta or triangular distributions are often used in project management), or derived from historical data.

158

D. Trastour and A. Christodoulou

4.2 Portfolio Valuation We now define the criteria that qualify the ‘best’ portfolio in a way that takes into account the uncertainty of the input data. Service Metrics. We assume that we have available a default forecast of the metrics being considered. This default forecast assumes that no changes are being done, i.e. no projects are selected and the level of budget and resources for operations stays the same. This forecast could be extracted from a domain expert or calculated using forecasting techniques [24]. We define this forecast as a random variable and note , the forecast for metric at time . As part of the demand management process, requests for IT resources need to be justified by a business case, and often quantifiable business impact is required by portfolio managers. Hence, we assume that we have estimations of the impact of various investments to the business service metrics. For instance reducing the shared database service support staff by 20% is expected to increase by 30 minutes the MTTR of the two services dependant on the shared database asset. This impact can be estimated by a domain expert or determined through capacity and demand management. Simulation methods have also been successful in modelling IT support organisations and in predicting the performance of the incident management process [10]. For generality in this paper we define the random variable , , to be the impact on metric of choosing the operations investment for asset . In a similar way, we also define , , for projects. Note that the impact of a project is time dependent, since it requires for a project to be completed, at least partially, to have some impact. The overall impact of a portfolio selection on metric is hence composed of the default forecast, the impact of all operations investments on all dependant assets, and the impact of projects: ,

,

,

.

, ,

,

.

, ,

(1)

Costs and resources. Cost and resource usage of projects and operational activities are also assumed to be present. Most IT organisations require this information in order to do portfolio selection. However the quality of estimations can vary significantly. We hence introduce the random variables , , and , , , respectively for costs and resource demands of asset operation investments, and , and , , for the corresponding demands of projects. Because when a project is submitted it has not been decided when it would start, the time index for , and , , is relative to the start of the project. The time index for , , and , , , is absolute. In addition, projects have the possibility of impacting the cost and resource demand of assets, for instance when introducing a new technology, and may have positive or negative consequences. We note , , the random variable modelling the impact of project on the cost of asset and , , , its impact on resources. Hence the total cost of running asset is:

Towards Robust IT Service Portfolio Management

,

.

,

, ,

,

.

(2)

, ,

A similar equation can be written for the resource requirement

159

, ,

.

Portfolio Valuation. We conclude this sub-section by giving the criteria for the service portfolio selection. For each metric , a decision maker can define a suitable target region , within ,, , , , . We can measure the likelihood of having , by calculating: , , ,

,

(3)

, ,

We can calculate in a similar way the likelihood of meeting budget and resource constraints. For simplicity we consider all random variables to be independent and can hence calculate the sum using the convolution of the variables for positive impacts, or the cross-convolution for negative impacts. When appropriate, one may want to also consider the sum or the average of certain metrics over time before calculating the likelihood. For instance, providing quarterly and yearly predictions for revenue may be valuable to a decision maker. 4.3 Generation and Interactive Selection of Service Portfolios IT executives often have multiple, possibly conflicting, objectives to take into account. We hence propose to model the problem as a multi-objective combinatorial optimisation problem [20] and aim at generating the set of feasible Pareto-efficient portfolios. The variables of the problem are the , and , defined above. The problem consists of maximise the set of objectives: ,

1,

,

1,

(4)

Because the problem is NP-complex, exact solutions may not be obtained for problems of realistic dimensions. But many meta-heuristics are available to approximate the Pareto-frontier [20]. A drawback of existing model-based optimal portfolio selection is that they use a ‘black-box’ approach. The solution is generated without leaving the decision makers the opportunity to make adjustments to the solution. Because all information may not have been modelled and because many stakeholders may be involved in the decision process, we believe a portfolio selection tool should be focused on decision support rather than decision making. When dealing with a small number of objective functions, in [19] the authors propose an effective user interface that allows a decision maker to iteratively refine her set of portfolio. As a next step, we will investigate techniques to visually explore the set of feasible and efficient service portfolios, possibly extending the ideas presented in [19] to support a larger set of objectives and to include the notion of likelihood.

160

D. Trastour and A. Christodoulou

5 Case Study and Validation To demonstrate the feasibility of our approach, we apply it to a simple example of an online PC retailer. We first describe the scenario and validate our models and methods by solving the service portfolio selection for the yearly IT planning. The business depends on 3 services: sales ( ), shipping ( ) and customer support. ( ). The set of IT assets that each service depends on are depicted in Fig. 1 Fig. 2. All services are evaluated by 3 metrics. The Revenue is in $1000s. The remaining two metrics evaluate the quality of service, and are averaged over the 3 services. Mean time between failures (MTBF) is measured in days, and mean time to recovery (MTTR) in minutes. For the purpose of this example, we assume a single, homogenous human resource pool, and ignore attributes such as skills, organisation, and geography that must be taken into account in real-life models. Based on market predictions, the expected revenue and standard deviation are forecasted to be 1000, 100 , 1000, 120 , 800, 150 , 1300, 200 for all 4 quarters. For simplicity, all random variables in this example are assumed to be normally distributed. Based on historical information and if no changes are made to the IT budget, organisation and infrastructure, the MTBF is expected to be identical for all 4 quarters and normally distributed with 14, 5 , and the MTTR is normally distributed with 300, 10 1. The IT organisation of this firm has set itself three objectives for the end of the year: (1) to increase its revenue by 10%, (2) to reduce its IT costs by 5% and (3) to improve customer satisfaction (measured by a two-fold improvement in MTBF and a 50% reduction in MTTR).

Fig. 2. Metrics, services and assets of the online PC retailer

A number of options for the operations of the IT assets have been proposed by the IT managers, in coordination with the business leaders. For each asset , two operation investment options exist: the minimum 1, and the ideal 2. To keep this example brief, we assume that operational costs and resource requirements are the 1

The exponential distribution for MTBF and the log-normal distribution for the MTTR would have been better choices. We have settled on the normal distribution here for ease of calculation of the convolution and cross-convolution.

Towards Robust IT Service Portfolio Management

161

Table 1. Subset of the operations investments Asset a=1 a=2

Option o=1 o=2 o=1 o=2

,

100 200 200 250

,

40 70 40 50

, ,

(revenue) 50, 0 0 50,

10

10

, ,

(MTBF) 3, 3, 0 3,

1 1 1

, ,

(MTTR) 10, 0 0 10,

2

2

same in every budgeting quarter , and we will note them , and , . Table 1 presents the investment options for the web farm and the database service. In addition to the operational investment, one or more projects can be invested in: increasing the shared database service capacity ( 1 , upgrading the web farm infrastructure ( 2 , implementing a new product search feature in the PC sales application ( 3 , implementing a build-your-own PC feature ( 4 , implementing an order tracing feature in-house ( 5 , and contracting out the order tracing functionality ( 6 . The investments in the database and web farm assets have reasonably high initial costs, but low labour requirements and relatively low uncertainty. Labour demands and costs are highly uncertain for the remaining projects that invest in new feature development with the exception of 6 that proposes to contract out the development and has minimum labour demands. The first two projects have impact mostly on the MTBF and MTTR metrics, as well as on the cost and staffing requirements of the web farm and shared database service assets. All other projects mostly affect the revenue. Finally, dependencies exist between the projects. If any of 3 or 4 is selected, more web capacity needs to be provided ( 2). Two exclusive options ( 6) and ( 7) are also provided for the “order tracking” feature. We have implemented the proposed solution using the JMetal framework for multi-objective optimisation [25]. Although an exhaustive enumeration algorithm would work on our small example, our goals are to tackle real-world problem of 100s to 1000s of investments. We chose to model the problem using the following variables: 1 binary string of length used to determine which projects are selected, integer variables to determine the start time of each project (index ), and integer variable to select operations investment for each asset (index ). Since this configuration is not a standard problem type, we have adapted the crossover and mutation operators to fit our data structures. We defined the target region for each objective to be within 10% of the expected value. When running the algorithm on the online PC retailer example, the algorithm gave a choice of 12 portfolios with 80% likelihood to meet all objectives (out of 131072 possible solutions).

6 Conclusion and Further Work We have presented a conceptual solution for IT service portfolio selection, an extension of project portfolio selection that also includes operational costs and resources.

162

D. Trastour and A. Christodoulou

We have formalised the problem by presenting an information model and a mathematical formulation. In particular, we have taken into account the uncertainty of the data when defining the objective functions. Finally, we have seen how the model could apply to a small online PC retailer example. We have demonstrated the feasibility of the approach on this small example by generating the Pareto-frontier of efficient portfolios. This paper gave an insight to the problem of IT service portfolio selection and an indication to the challenging requirements that solutions must meet. While our preliminary results are encouraging, more work needs to be done regarding the validation of the approach. Our immediate next steps are to test the scalability of the solution to understand its limits. Important issues such as the modelling of uncertainty, the nature of underlying distributions, the complexity of assets, services and metrics, are not addressed here and will be the subject of further work. Finally, we will research techniques to visually explore the trade-offs that decision makers have to make. The effectiveness and robustness of the solution will rely on human decision makers so an interactive, real-time, user-friendly solution is necessary.

References 1. Office of Government Commerce: The Introduction to the ITIL Service Lifecycle Book. TSO (2007) 2. Symons, C.: Service Portfolio Management Links IT Capabilities To Business Value. Forrester (2008) 3. Office of Government Commerce: Service Strategy Book. TSO (2007) 4. Moura, A., Sauve, J., Bartolini, C.: Research Challenges of Business-Driven IT Management. In:Proceedings BDIM 2007, pp.19–28 (2007) 5. Reboucas, R., Sauve, J., Moura, A., Bartolini, C., Trastour, D.: A decision support tool to optimize scheduling of IT changes. In: Proceedings of 10th IFIP/IEEE IM 2007, May 2007, pp. 343–352 (2007) 6. Trastour, D., Rahmouni, M., Bartolini, C.: Activity-Based Scheduling of IT Changes. In: Bandara, A.K., Burgess, M. (eds.) AIMS 2007. LNCS, vol. 4543, pp. 73–84. Springer, Heidelberg (2007) 7. Aiber, S., Gilat, D., Landau, A., Razinkov, N., Sela, A., Wasserkrug, S.: Autonomic SelfOptimization According to Business Objectives. In: Proceedings of ICAC 2004, New York, NY, USA, May 17-19, pp. 206–213 (2004) 8. Sauvé, J., Marques, F., Moura, A., Sampaio, M., Jornada, J., Radziuk, E.: SLA Design from a Business Perspective. In: Proceedimgs of 16th IFIP/IEEE DSOM 2005, Barcelona, Spain, October 24-26, p. 72–83 (2005) 9. Boutaba, R., Xiao, J., Aib, I.: CyberPlanner: A comprehensive toolkit for network service providers. In : Proceedings of IEEE/IFIP NOMS 2008, April 7–11, Salvador, Bahia, Brazil, pp.379–386 (2008) 10. Bartolini, C., Stefanelli, C., Tortonesi, M.: SYMIAN: A Simulation Tool for the Optimization of the IT Incident Management Process. In: De Turck, F., Kellerer, W., Kormentzas, G. (eds.) DSOM 2008. LNCS, vol. 5273, pp. 83–94. Springer, Heidelberg (2008) 11. Markowitz, H.: Portfolio selection. Journal of finance, 77–91 (1952) 12. Sharpe, W.: A simplified model for portfolio analysis. Management science, 277–293 (1963) 13. Martino, J.: R&D Project Selection. Wiley, New York (1995)

Towards Robust IT Service Portfolio Management

163

14. Golab, K., Kirkwood, C., Sicherman, A.: Selecting a portfolio of solar energy projects using multiattribute preference theory. Management Science, 174–189 (1981) 15. Souder, W., Mandakovic, T.: R&D project selection models. Research Management 29(4), 36–42 (1986) 16. Saaty, T.: The analytical hierarchy process. McGraw-Hill, New York (1980) 17. Chen, J., Askin, R.: Project selection, scheduling and resource allocation with time dependent returns. European Journal of Operational Research 193(1), 23–34 (2009) 18. Heidenberger, K., Stummer, C.: Research and development project selection and resource allocation: a review of quantitative modelling approaches. International Journal of Management Reviews 1(2), 197–224 (1999) 19. Stummer, C., Heidenberger, K.: Interactive R&D portfolio analysis with project interdependencies and time profiles of multiple objectives. IEEE Transactions on Engineering Management 50(2), 175–183 (2003) 20. Ehrgott, M., Wiecek, M.: Multiobjective Programming, pp. 667–722. Springer, Boston (2005) 21. Steuer, R.: Multiple criteria optimization: Theory, computation, and application. John Wiley & Sons, Chichester (1986) 22. Liesiö, J., Mild, P., Salo, A.: Robust portfolio modeling with incomplete cost information and project interdependencies. European Journal of Operational Research 190(3), 679–695 (2008) 23. IT Governance Institute: COBIT 4.1. ISA (2007) 24. Jenkins, G.: Time Series Analysis, Forecasting, and Control. Holden-Day (1976) 25. Durillo, J., Nebro, A., Luna, F., Dorronsoro, B., Alba, E.: jMetal: A Java Framework for Developing Multi-Objective Optimization Metaheuristics., Departamento de Lenguajes y Ciencias de la Computación, University of Málaga (2006)

Hidden Markov Model Modeling of SSH Brute-Force Attacks Anna Sperotto, Ramin Sadre, Pieter-Tjerk de Boer, and Aiko Pras University of Twente Centre for Telematics and Information Technology Faculty of Electrical Engineering, Mathematics and Computer Science P.O. Box 217, 7500 AE Enschede, The Netherlands {a.sperotto,r.sadre,p.t.deboer,a.pras}@utwente.nl

Abstract. Nowadays, network load is constantly increasing and high-speed infrastructures (1-10Gbps) are becoming increasingly common. In this context, flow-based intrusion detection has recently become a promising security mechanism. However, since flows do not provide any information on the content of a communication, it also became more difficult to establish a ground truth for flowbased techniques benchmarking. A possible approach to overcome this problem is the usage of synthetic traffic traces where the generation of malicious traffic is driven by models. In this paper, we propose a flow time series model of SSH brute-force attacks based on Hidden Markov Models. Our results show that the model successfully emulates an attacker behavior, generating meaningful flow time series.

1 Introduction Since the last decade, we are facing a constant rise of both network load and speed. This has become a problem for packet-based network intrusion detection systems since a deep inspection of the packet payloads is often not feasible in high-speed networks. In this context, flow-based intrusion detection emerged as an alternative to packet-based solutions [1]. Dealing with aggregated network measures, such as flows, helps in reducing the amount of data to be analyzed. A flow is defined as “a set of IP packets passing an observation point in the network during a certain time interval and having a set of common properties” [2]. A flow carries information about the source/destination IP addresses/ports involved in the communication, but nothing is known about the content of the communication itself. Flow-based time series are a well-known way of visualizing network information [3]. Flow-based time series allow data processing in a streaming fashion, offer a compact representation of network traffic and allow to analyze data keeping into account the temporal relations between events. The drawback of flow-based time series is that we cannot have direct evidence of when an attack happened. In most cases, a ground truth is missing. Attack-labeled flow data sets are rare and their creation is a lengthy and time consuming process [4]. To overcome this problem, approaches based on the superposition of real non-malicious traffic with synthetic attack traffic have been introduced [5,6]. For these approaches to work, we need models of network attacks. C. Bartolini and L.P. Gaspary (Eds.): DSOM 2009, LNCS 5841, pp. 164–176, 2009. c IFIP International Federation for Information Processing 2009

Hidden Markov Model Modeling of SSH Brute-Force Attacks

165

In this paper, we propose a time-series based model of SSH brute-force attacks built upon the concept of Hidden Markov Models. We show that our model is able to emulate important aspects of the network behavior of such attacks and generates meaningful flow-based traffic time series. This paper is organized as follows. Section 2 summarizes the related work on modeling of malicious traffic. Section 3 describes how SSH brute-force attacks are characterized at flow level. Section 4 presents our model for SSH malicious traffic. The model is evaluated in Section 5. Finally, conclusions are drawn in Section 6.

2 Related Work Hidden Markov Models (HMM) are effective in modeling sequential data [7]. Since they have been introduced in the early 1970s [8], they have been successfully applied to different scientific fields. Examples are biological sequence analysis [9], speech recognition [10] and pattern recognition [11]. Hidden Markov Model triggered the curiosity of many researchers also in the fields of Networking and Intrusion Detection. HMM can be trained on real data and their main characteristic is the ability to capture the temporal behavior of the observed processes. The work of [12] proposes to formalize traffic exchange in terms of “HMM profiles”, a stochastic structure suited for sequence alignment. The results show that the models are able to classify traffic sequences at application level. In [13,14], the authors propose a packet-level model of traffic sources based on HMM. The model proves to be effective in application classification. Moreover, a second fruitful application of the model in [13,14] is in traffic prediction, namely forecast of short-term future traffic behavior. Similarly to these last contributions, we will use our model for traffic generation. Nevertheless, the work in [12,13,14] focuses on the packet-level, while we are interested in flow-based time series. Hidden Markov Models are particularly appealing in Intrusion Detection, since they are able to calculate how likely a certain sequence of events is. Behavioral models for host-based intrusion detection have been proposed in [15] and [16]. The authors profile the normal sequence of system calls and raise alarms whenever a sequence is unlikely to be seen. Contrary to these contributions, we model malicious activities based on network data. A different approach is the one in [17], where the hidden states model the safety status of a network. Similarly to them, we also assigned semantic meaning to the hidden states. However, in our case the hidden states model the behavior of an attacker. SSH brute-force attacks are a well known cyber threat [18,19]. Although the attack is very common, it is still potentially dangerous. Our studies [4] showed that newly set up vulnerable hosts can be compromised within few days and be used as platform for the same attacks. We also showed that SSH attacks are visible at flow level as peaks in the SSH flow time series [20]. However, this observation tells us only how SSH attacks affect the total network traffic when such attacks are at their peak of intensity. In this paper we want to explore how the entire time series generated by a single attacker evolves in time.

166

A. Sperotto et al.

3 Flow-Based Characterization of SSH Brute-Force Attacks

flows

450 400 350 300 250 200 150 100 50 0

packets / 10

700 600 500 400 300 200 100 0

bytes / 103

Brute-force SSH attacks are one of the most common threats in cyber space [19]. In this section we qualitatively characterize such attacks at flow level, namely describing what a brute-force SSH attack looks like if only flow information is available. We analyze the traffic generated by a host known to have performed a SSH brute-force attack against our university network. The attack took place in the early afternoon of July 16, 2008 and lasted approximately 40 minutes. During this interval, approximately 8300 distinct university hosts have been attacked. The attack generated a volume of traffic of approximately 32,400 flows, 279,000 packets and 30.5MB. Figure 1 shows for the attacker the evolution over time of (i) the number of flows created per second, (ii) the number of packets transferred per second, and (iii) the number of bytes transferred per second. The time resolution of the time series is 1 second. Each value in the time series accounts for both the traffic generated by the attacker and the traffic that he receives from the victims. In this way, it is possible to characterize in the same time series the entire attack. During the attack, its intensity varies. In the flow time series we can see that in about the first 1000 seconds the attack intensity grows, reaching a peak of 450 flows/s. After that, the number of flows per second drops abruptly and roughly stabilizes around 100 flows/s. Finally, the attack activity slowly fades off in the last 500 seconds. Moreover, a deeper analysis of the flow time series shows that the activity pattern is not constant in time: each second of activity is often followed by one or more seconds of inactivity. The packet and byte time series closely follow the one of flows. Both show a peak around 1000 seconds since the beginning of the attack. As for flows, the trend of packets and bytes tends to stabilize for a while before the attack slowly dies. This behavior suggests that during an SSH brute-force scan, the flows, packets and bytes statistics are mutually correlated.

400 350 300 250 200 150 100 50 0

0

500

1000

1500

2000

2500

3000

2000

2500

3000

2000

2500

3000

Time (s)

0

500

1000

1500 Time (s)

0

500

1000

1500 Time (s)

Fig. 1. Flow, packet and byte time series for a malicious SSH user

Hidden Markov Model Modeling of SSH Brute-Force Attacks

167

70000 FROM ATTACKER TO ATTACKER 60000

50000

IP

40000

30000

20000

10000

0 0

500

1000

1500 Time (s)

2000

2500

1500

2000

2500

3000

(a) 16

14

12

ppf

10

8

6

4

2

0 0

500

1000

3000

Time (s)

(b) Fig. 2. Temporal visualization of a brute-force SSH scan (a) and variation of packets per flow during the scan (b)

A different view on the attacks is given by Figure 2(a). Each mark in the graph either represents a malicious connection from the attacker to a victim or the answering connection from the victim back to the attacker. The y-axis gives the 65,535 possible destination addresses in the university network. We identify three attack phases. During the scanning phase (first 1000 seconds), the attacker performs a sequential SSH scan spanning over the entire network address space. In this phases, the attacker gathers information on which hosts run a vulnerable SSH service. Only few victims respond to the attack. Once this phase is completed, the attacker initiates a brute-force user/password guessing attack (brute-force phase). In this phase, only a small subset of the hosts in the network is involved. This phase corresponds to the second block of 1000 seconds

168

A. Sperotto et al.

and is characterized by a high interaction between the attacker and the victims. Finally, after about 2000 seconds since the beginning of the attack, the brute-force phase ends. Nevertheless, the time series in the previous picture have already shown that after this moment in time there is still traffic. Form Figure 2(a) it is now evident that the residual traffic is due to compromised hosts that communicate with the attacker. We refer to this final phase as the die-off phase. Although the three attack phases are clearly visible in Figure 2(a), they are not so clearly identifiable from the flow, packet and byte time series shown in Figure 1. However, the fact that the three time series are correlated allows us to derive a more suitable measure. Figure 2(b) shows the evolution over time of the packets per flow, again with a resolution of 1 second. Using this measure, the three phases are clearly visible. The scanning phase is characterized by only few packets per flow, in average between 1 and 1.5. These values are consistent with a scenario in which several three-way handshakes are initiated but only few are completed. When the brute-force phase starts, the number of packets per flow has a sharp rise: from 1.5 to a average of about 11. During this phase, several user/password combinations are tested against the same victim. This explains why the attacker produces a higher number of packets per flow. Finally, the die-off phase sees again only few packets per flow. In the majority of the cases, we observe only one packet per flow. The variation of packets per flow over time seems therefore to be a key characteristic of the behavior of an SSH brute-force attacker. It moreover shows that the flow, packet and byte time series still carry enough information to characterize the attack. The attack behavior that we described in this section is typical for a brute-force SSH attack. While monitoring the university network, we observe in average one attack with these characteristics per day. In the following section, we will describe how the flowbased characterization that we just proposed can be used to model the behavior of an SSH brute-force attacker.

4 Modeling with HMM The analysis of a typical SSH brute-force attack that we presented in Section 3 pointed out three key characteristics of such attacks: 1. The flow, packet and byte time series exhibit a clear correlation. 2. Attacks consist of three phases: a scanning phase, a brute-force phase and a die-off phase. 3. The subdivision into phases may not be evident when we observe the flow, packet and byte time series directly, but it becomes manifest when we consider the packet per flow time series. These key characteristics will play a central role in the following when modeling attacks using Hidden Markov Models (HMM). This section is organized as follows. Section 4.1 briefly recapitulates the definition of HMM. In Section 4.2 we show how an SSH brute-force attack can be described as a Markov Chain. In Section 4.3 we describe the output probabilities associated to each state and how they can be used to generate meaningful time series. Finally, Section 4.4 explains how the model parameters are computed from real data traces.

Hidden Markov Model Modeling of SSH Brute-Force Attacks

169

4.1 Hidden Markov Models Hidden Markov Models are a class of statistical models able to describe sequences of data resulting from the interaction of several random processes. Formally, an HMM is a discrete time Markov chain (DTMC) where each state is augmented with a probability distribution over a finite set of output symbols. Given a sequence of states Q = q1 q2 . . . with associated output symbols K = k1 k2 . . . we say Q forms the hidden sequence and K forms the observation sequence. With S = {s1 , . . . , sn } we denote the finite set of hidden states. S is called hidden chain. With qt we denote the state at time t. With aij we denote the probability of jumping from state si to state sj . Since we are dealing with a DTMC, this probability only depends on the current state si , i.e.: aij = P (qt+1 = sj | qt = si ). With πi we denote the probability of the initial state being si , i.e.: πi = P (q1 = si ). With O = {o1 , . . . om } we denote the finite set of output symbols. With kt we denote the output symbol seen at time t. With bi (o) we denote the probability of seeing output symbol o when the hidden state is si , i.e.: bi (o) = P (kt = o | qt = si ). An HMM separates the state chain from the observable output. This key characteristic allows us to model malicious SSH attacks in an effective way and to generate synthetic flow, packet and byte time series. 4.2 The Hidden Chain In Section 3, we explained that an SSH brute-force attack consists of three phases: a scanning phase, a brute-force phase and a die-off phase. We make use of these phases to define the hidden chain. Our model consists of the following seven states: – the states Si , i = 1, 2, 3. In these states, the attacker is active and causes network traffic. – the states Ii , i = 1, 2, 3. In these states, the attacker is temporary inactive, as described in Section 3. – the end state End.

0.004 0.617

start

0.378

S1

0.813

I1

0.04 0.001

0.723

0.235

S2

0.186

0.757

0.029

I2 0.214

0.667

0.269

S3 0.022

0.848

I3 0.152

0.039

0.001

0.003 0.001

End

Fig. 3. HMM for the SSH brute-force attack with an example of transition probabilities learnt from real data

170

A. Sperotto et al.

The state S1 is the start state with πS1 = 1. Figure 3 depicts the states and the possible transitions. The states S1 and I1 model the scanning phase of the attack. As it can be seen in Figure 3, once the attack moves from the scanning phase to the brute-force phase, represented by the states S2 and I2 , it cannot return to the previous phase. This ensures that the scan will not be performed more that once for each attack. On the other hand, the die-off phase (states S3 and I3 ) can partially overlap with the brute-force phase. This phenomenon is modeled by making the states {S2 , I2 , S3 } a fully connected chain. However, the transition probabilities for this subset of states will privilege transitions in the same phase. Finally, the state End models the end of an attack. We allow the model to jump from each active state Si to the End state, thus reflecting the fact that some attacks stop after the scan phase or the brute-force phase. 4.3 The Output Probabilities The aim of our model is to generate meaningful synthetic flow, packet and byte time series for a SSH brute-force attack. Hence, at each transition, our model should output a triple (F, P, B) with the values for the three time series. It is important to note that these three values are not independent, as shown in Section 3. Hence, to generate correctly correlated values for the three time series, a joint output probability distribution PF,P,B would be needed for each state of the model. In the following, we will present a different approach that approximates the triple-joint probability distribution PF,P,B . To each active state Si , i = 1, 2, 3 in our model we assign the following two distributions: 1. an empirical probability distribution of flows PF ; 2. an empirical joint probability distribution of packets per flow (P P F ) and bytes per packet (BP P ), denoted as PP P F,BP P . At each transition, random values of F , P P F and BP P are generated according to the empirical distributions associated to the current state. Given the number of flows F , we assume the number of packets per flows and the number of bytes per packets to be the same for all the flows for this emission. We calculate P = P P F · F, B = BP P · P P F · F. The joint probability distribution PP P F,BP P and the indirect computation of P and B by the above expression ensure the strong correlation between F , P and B that we have observed in the data. In the states Ii , i = 1, 2, 3, the attacker is by definition temporarily inactive and the triple (0, 0, 0) is the only allowed output. 4.4 The Parameter Estimation Once the hidden chain and the outputs of the model have been defined, we need to estimate the transition probabilities and the emission probability distributions for the states

Hidden Markov Model Modeling of SSH Brute-Force Attacks

171

Si , i = 1, 2, 3. Several methods for estimating the parameters of an HMM have been proposed in literature, for example the Baum-Welch algorithm [8], or the simulated annealing method of [21]. However, these methods are used when the training is based on sequences of observations only and the hidden state sequence is unknown. In our training procedure, we follow a different approach. The analysis of the packets per flow time series, such as the one in Figure 2(b), offers us a way to precisely relate each observation in a trace with the hidden state that emitted it. We therefore manually labeled the traces in our training data sets. Once the hidden state sequence is known, we calculate each transition probability as aij =

|{transitions from si to sj }| . |{transitions from si }|

Figure 3 gives an example of transition probabilities learnt from real data. The hidden state sequence is used to compute the output probabilities associated to each state. We calculate the distributions PF and PP P F,BP P for a state Si , i = 1, 2, 3 from the frequency histograms of the observations emitted from that state.

5 Validation In this section we will evaluate the performance of the model proposed in Section 4. In particular, we will show that the model is able to generate synthetic traffic that has the same statistical characteristics of the SSH brute-force attack traces we used as training. We based our validation on two data sets consisting of malicious SSH traces collected at the University of Twente network (original data sets). The validation proceeds as follows. First, we train an HMM for each distinct data set. Second, we use the models to generate groups of synthetic traces sufficiently large for the calculation of the confidence intervals. We refer to these traces as synthetic data sets. Third, we analyze the statistical properties of the synthetic data sets and we compare them with the original data sets. The aim of this analysis is to show that the model is able to encode sufficient information to correctly emulate the original traces. Section 5.1 describes the data sets used for the training, while Section 5.2 explains how the synthetic traces are generated. Section 5.3 introduces our testing methodology. Finally Section 5.4 presents our results. 5.1 Original Data Sets Our model of SSH brute-force attacks has been tested on two data sets. Each data set contains flow, packet and byte time series for a group of hosts known to have scanned our networks. The malicious hosts are all distinct. The time series have been created considering time slots of 1 second. The volume of traffic for each time slot is comprehensive of both the traffic generated by the scanner and the traffic that it receives. Table 1 presents the data sets. Both data sets have been collected during a monitoring window of one week on the network of the University of Twente. The offending hosts have been identified by a high interaction honeypot that is normally active in our

172

A. Sperotto et al. Table 1. Statistical characteristic of the collected data sets Data Set Collection time Traces Avg. Flows/sec Avg. Packets/sec Avg. Bytes/sec Set 1 13-20 July 2008 17 11.06 66.91 7337.33 Set 2 19-26 April 2009 13 15.80 150.52 19016.00

network. Set 1 has been collected in July 2008 and consists of 17 traces. Set 2 has been collected in April 2009 and consists of 13 traces. Other hosts performing SSH malicious activities have not been considered part of the data sets since they appear to belong to a different class of scans. The statistical analysis of the two data sets shows that the average values of flows, packets and bytes over time have changed in time. In Set 2, the attackers appear to produce in average more that twice the amount of packets and bytes compared to Set 1. This suggests that, while the attack mechanism stays the same, the attacks’ intensities are likely to vary in the course of time. As a consequence, models trained on real data would need periodical retrain. 5.2 Synthetic Trace Generation We define a synthetic trace as the sequence of observations that the model outputs when a random path is taken. The generation process can be summarized as follows. Let’s assume the model to be in state si : 1. at time t, the model jumps from the current state si to the next state sj according to the transition probabilities aij , j = 1, . . . n. 2. if sj is the End state, the path is concluded and the trace ends. 3. once sj has been selected, the model randomly selects F , P P F and BP P . 4. the model outputs the triple (F, P, B), calculated on the basis of the random values generated in the previous step (as explained in Section 4.3). 5. once the observations have been emitted, the process iterates from step 1. At each iteration, the model chooses which triple (F, P, B) will be emitted. This choice is independent from the previous outputs and is controlled only by the empirical probability distributions of F , P P F and BP P associated with the current state. Table 2 presents the range of these distributions for both Set 1 and Set 2. The model controls also the duration of a trace, since a trace ends only when a transition to the End state is randomly selected. 5.3 Testing Methodology Our testing methodology aims to measure the average statistical characteristics of a set of synthetic traces and compare them to the ones of the original data sets Set 1 and Set 2. Each statistical metric is calculated for flows, packets and bytes. We are interested in three types of statistical measures: – the mean and standard deviation for flow (μF , σF ), packets (μP , σP ) and bytes (μB , σB ). These measures describe the overall behavior of flows, packets and bytes independently of each other in a trace.

Hidden Markov Model Modeling of SSH Brute-Force Attacks

173

Table 2. Empirical distribution ranges for the training data sets Distribution F phase 1 F phase 2 F phase 3 P P F phase 1 P P F phase 2 P P F phase 3 BP P phase 1 BP P phase 2 BP P phase 3

Set 1 Min Max 1 789 1 519 1 227 1 26.4841 1 16.5 1 5 40 156.42 50.88 267.27 40 836

Set 2 Min Max 1 3825 1 860 1 250 1 27 1 17 1 5 40 225.71 52 319.42 46 1148

– the correlation coefficients ρF P , ρF B and ρP B between flows, packets and bytes. These measures describe the dependence between flows, packets and bytes in the same trace. – autocorrelation of lag 1 of flows (RF ), packets (RP ) and bytes(RB ). The autocorrelation captures the evolution of a trace over time, measuring the interrelation of the trace with itself in different moments in time. The previously introduced measures are relative to a single trace. In our experimental results, we calculate the average values of each measure for both the original data sets and the synthetic ones. For the synthetic trace, we also calculate the 95% confidence intervals. Each synthetic data set consists of 300 traces. Finally, we evaluate how well the synthetic traces approximate the original ones. In order to do so, we calculate for each measure m the relative error between the original traces and the synthetic ones: Err =

|morig − msyn | morig

5.4 Experimental Results This subsection presents the numerical results obtained from the analysis of the synthetic data sets. Table 3 offers an overview of the average statistical measures for both the original and the synthetic data sets. The same table also lists the relative error between original and synthetic measures. The results will be discussed in the following sections. Average mean and standard deviation. Both the model trained on Set 1 and the one trained on Set 2 approximate the averages of flows, packets and bytes within a 10% relative error. The results also show that our approach approximates the standard deviation of both the original data sets within 10% relative error, with only few exceptions: the average standard deviation of packets for Set 1 and the average standard deviations of flows and bytes for Set 2. Regarding Set 1, the synthetic measure underestimates the one in the original data set. On the contrary, in Set 2, the synthetic measures are higher than the

174

A. Sperotto et al. Table 3. Numerical comparison between the original and the synthetic data sets Set 1 Synthetic 1 μF 11.06 12.27 ± 0.33 μP 66.91 66.66 ± 3.67 μB 7337.33 7524.73 ± 523.11 σF 36.45 38.33 ± 1.12 σP 324.29 243.43 ± 10.91 σB 28510.35 28345.60 ± 1616.63 ρF P 0.79 0.79 ± 0.012 ρF B 0.76 0.74 ± 0.016 ρP B 0.94 0.98± 0.002 μRF 0.46 0.23 ± 0.009 μRP 0.56 0.25 ± 0.012 μRB 0.58 0.26 ± 0.012

Err Set 2 Synthetic 2 0.109 15.80 15.15 ± 0.65 0.046 150.52 138.85 ± 8.5 0.025 19016.00 18107.88 ± 1153.53 0.051 40.0 47.01 ± 1.87 0.249 379.38 419.16 ± 16.55 0.005 47060.07 55378.58 ± 2239.91 0.001 0.83 0.86± 0.01 0.023 0.79 0.81± 0.01 0.047 0.98 0.98± 0.001 0.498 0.64 0.26 ±0.01 0.547 0.71 0.30 ± 0.009 0.549 0.75 0.30 ± 0.009

Err 0.041 0.077 0.047 0.174 0.104 0.176 0.039 0.024 0.001 0.593 0.577 0.592

original. We suspect this phenomenon is related to the autocorrelation of the traces in the original data sets. Table 3 also presents the 95% confidence intervals for the average means and the standard deviations. For all measures, the confidence intervals are close to the average values. Average correlation. The correlation coefficients show that the proposed model is able to capture the interrelations between flows, packets and bytes, despite that the realizations of the random variables F and P P F are independently drawn. The relative error in the case of the correlation coefficients is indeed less or equal to 4.7% (ρP B ) in the case of Set 1 and less or equal to 3% (ρF P ) in the case of Set 2. In the same table we listed also the 95% confidence intervals for the average correlation coefficients. As in the case of the average relative error, described in the previous section, the confidence intervals are closed to the mean values. Average autocorrelation. The last measure we consider is the average autocorrelation. The autocorrelation characterizes the temporal evolution of a trace. For both Set 1 and Set 2 our model fails to approximate the autocorrelation values. The autocorrelation of the synthetic traces, indeed, is roughly half of the autocorrelation in the original data sets. This means that consecutive values in a synthetic trace have a higher random component than in the original traces. We believe that the cause of lower autocorrelation coefficients can be found in the attacker behavior during the brute-force phase. The original traces, indeed, show that during this phase the time series presents a certain regularity, as for example a bounded number of flows per seconds. Our model, on the other hand, randomly selects at each iteration new values for flows, packets and bytes, without any memory of the previous outputs. This behavior is reflected in lower autocorrelation values. We consider to extend the model to capture regularity in the brute-force phase as a possible future work. As for the previous measures, also in this case the confidence intervals show that the model has a low variability in the autocorrelation values.

Hidden Markov Model Modeling of SSH Brute-Force Attacks

175

6 Conclusions In this paper, we have presented a compact model of SSH brute-force attacks based on Hidden Markov Models. The model has been inferred on the basis of only flow information and it encodes the network behavior of SSH attacks: scanning phase, bruteforce phase and die-off phase. The model parameters have been calculated on the basis of real data traces captured at the University of Twente network. In this paper we also demonstrate that the model, once trained on real data, is able to emulate the network behavior of a SSH brute-force attacker. Synthetic traces approximate the mean, standard deviation and correlation of flow, packet and byte time series within 10% relative error. The model fails only in approximating the autocorrelation. The synthetic traces, indeed, seem to have a higher random component than the original training trace. As far as we are aware, this was the first time that HMM have been applied to the generation of flow-based time series for malicious users. The results are encouraging, but many aspects are open for future work. First, we aim to refine the model. For example, a more detailed model of the brute-force phase can improve the autocorrelation. In addition, the empirical emission distributions can be substituted by estimated distribution functions to make the model resilient to unforeseen observations. Second, we plan to adapt the model to be used for detection. In this context, we are also interested in investigating if the model we proposed is suitable for detection of other brute-force attacks that show a similar phase behavior. An example can be a brute-force attack against the telnet service. Third, we want to apply our HMM approach to other attack types, such as DoS attacks or worms. Acknowledgments. This research has been supported by the EC IST-EMANICS Network of Excellence (#26854).

References 1. Sperotto, A., Schaffrath, G., Sadre, R., Morariu, C., Pras, A., Stiller, B.: An Overview of IP Flow-based Intrusion Detection. IEEE Communications Surveys & Tutorials (to appear, 2009) 2. Quittek, J., Zseby, T., Claise, B., Zander, S.: Requirements for IP Flow Information Export (IPFIX). RFC 3917 (Informational) 3. NfSen - Netflow Sensor (May 2009), http://nfsen.sourceforge.net 4. Sperotto, A., Sadre, R., van Vliet, D.F., Pras, A.: A Labeled Data Set For Flow-based Intrusion Detection. In: Nunzi, G., Scoglio, C., Li, X. (eds.) IPOM 2009. LNCS, vol. 5843, pp. 39–50. Springer, Heidelberg (2009) 5. Brauckhoff, D., Wagner, A., Mays, M.: FLAME: a flow-level anomaly modeling engine. In: Proc. of the Workshop on Cyber Security Experimentation and Test, CSET 2008 (2008) 6. Sommers, J., Yegneswaran, V., Barford, P.: A framework for malicious workload generation. In: Proc. of the 4th ACM SIGCOMM conference on Internet measurement, IMC 2004 (2004) 7. Camastra, F., Vinciarelli, A.: Markovian models for sequential data. Machine Learning for Audio, Image and Video Analysis (2008)

176

A. Sperotto et al.

8. Baum, L.E., Petrie, T., Soules, G., Weiss, N.: A Maximization Technique Occurring in the Statistical Analysis of Probabilistic Functions of Markov Chains. The Annals of Mathematical Statistics 41 (1970) 9. Durbin, R., Eddy, S.R., Krogh, A., Mitchison, G.: Biological Sequence Analysis: Probabilistic Models of Proteins and Nucleic Acids. Cambridge University Press, Cambridge (1998) 10. Rabiner, L.R.: A tutorial on hidden Markov models and selected applications in speech recognition. Proceedings of the IEEE (1989) 11. Fink, G.A.: Markov Models for Pattern Recognition: From Theory to Applications. Springer, New York (2007) 12. Wright, C.V., Monrose, F., Masson, G.M.: HMM Profiles for Network Traffic Classification. In: Workshop on Visualization and Data Mining for Computer Security VizSEC/DMSEC 2004 (2004) 13. Dainotti, A., Pescapé, A., Rossi, P.S., Palmieri, F., Ventre, G.: Internet traffic modeling by means of Hidden Markov Models. Computer Networks 52(14) (2008) 14. Dainotti, A., de Donato, W., Pescape, A., Rossi, P.: Classification of Network Traffic via Packet-Level Hidden Markov Models. In: Proc. of IEEE Global Telecommunications Conference, GLOBECOM 2008 (2008) 15. Gao, D., Reiter, M.K., Song, D.X.: Behavioral Distance Measurement Using Hidden Markov Models. In: Zamboni, D., Krügel, C. (eds.) RAID 2006. LNCS, vol. 4219, pp. 19–40. Springer, Heidelberg (2006) 16. Warrender, C., Forrest, S., Pearlmutter, B.: Detecting Intrusions Using System Calls: Alternative Data Models. In: Proc. of the 1999 IEEE Symposium on Security and Privacy (1999) 17. Khanna, R., Liu, H.: System approach to intrusion detection using hidden Markov model. In: Proceedings of the 2006 International Conference on Wireless communications and mobile computing, IWCMC 2006 (2006) 18. Seifert, C.: Analyzing malicious ssh login attempts (September 2006), http://www.securityfocus.com/infocus/1876 19. SANS Institute: Top-20 2007 Security Risks (2007 Annual Update), www.sans.org (May 2009) 20. Sperotto, A., Sadre, R., Pras, A.: Anomaly Characterization in Flow-Based Traffic Time Series. In: Akar, N., Pioro, M., Skianis, C. (eds.) IPOM 2008. LNCS, vol. 5275, pp. 15–27. Springer, Heidelberg (2008) 21. Andrieu, C., Doucet, A.: Simulated Annealing for Maximum A Posteriori Parameter Estimation of Hidden Markov Models. IEEE Transactions on Information Theory 46 (2000)

Network Virtualization in Future Home Environments Andreas Berl1 , Roman Weidlich2 , Michael Schrank1 , Helmut Hlavacs2 , and Hermann de Meer1 1

2

Computer Networks and Communications, University of Passau, Germany Institute of Distributed and Multimedia Systems, University of Vienna, Austria

Abstract. Home environments have a great potential of resource sharing and energy saving. More and more home computers are running on an always-on basis (e.g. media-centers or ﬁle-sharing clients). Such home environments have not been suﬃciently analyzed regarding their energy-eﬃcient operation, yet. This paper discusses network virtualization methods that are needed in future home environments to enable the energy-eﬃcient cooperation of home networks. End-users share their available hardware resources (e.g. CPU, disk, or network resources) with other users in an energy-eﬃcient and balanced way. To achieve such an envisioned future home environment, an architecture is suggested that combines diﬀerent virtualization methods. In this paper, virtualization related requirements of the suggested architecture are discussed in detail. Network virtualization methods and concepts are compared to each other with respect to their usability in the architecture. In addition, initial virtualization approaches are simulated and evaluated with regard to beneﬁts and complexity in the suggested architecture. Keywords: Home networks, energy eﬃciency, resource sharing, virtualization, peer-to-peer.

1

Introduction

Increased costs of energy and the desire to reduce CO2 emissions make energyeﬃcient computing a more and more important topic. Koomey [1] reports a doubling of energy consumption from 2000 to 2005 of volume, mid-range, and high-end servers in the U.S. and worldwide. Although this is related to data centers, a similar tendency can be expected for computers in home environments. End devices in the home are contributing to a large portion of the electricity consumption growth according to a 2006 survey commissioned by the EU [2]. Following energy-saving concepts of data-centers, an energy-eﬃcient Virtual Home Environment (VHE) architecture is suggested and evaluated in [3,4,5,6] and presented in Section 3. Energy wastage in data centers is mainly caused by underutilized hardware. To increase energy-eﬃciency, services can be virtualized and consolidated (several services run on the same hardware). The VHE architecture realizes this kind of consolidation in home networks. Hardware resources C. Bartolini and L.P. Gaspary (Eds.): DSOM 2009, LNCS 5841, pp. 177–190, 2009. c IFIP International Federation for Information Processing 2009

178

A. Berl et al.

of end-hosts (e.g. CPU cycles, disk space, or network capacity) are virtualized and shared between users in an energy eﬃcient way. Always-on services of users (e.g. media-servers or ﬁle-sharing applications) are consolidated on end-hosts and unused computers are turned oﬀ (or hibernated) to save energy. Although this concept seems to be very similar to energy-saving concepts of data centers, there are severe diﬀerences. Services in data centers are located in controlled closed environments with high bandwidth networks. Distributed home environments have no central management and require diﬀerent virtualization and management methods to share hardware resources and energy in a balanced way. To enable the envisioned hardware resource sharing within the VHE architecture, several virtualization related requirements have to be fulﬁlled: – Resource availability: Provision of idle hardware resources in separate runtime environments; – Home network interconnection: Addressing and locating of participating home networks; – Resource mediation: Addressing and locating of idle hardware resources; – Resource allocation: Distributed management of state and resource information; These virtualization related requirements and possible solutions for it are discussed in detail in this paper. Diﬀerent kinds of virtualization methods are needed to realize the envisioned energy-eﬃcient resource sharing among endusers. Especially network virtualization approaches are analyzed in this paper and virtualization approaches are suggested that meet the deﬁned requirements. Furthermore, network virtualization approaches are evaluated by simulation. Initial simulations and evaluations of the suggested VHE architecture have already been shown in [3,4,5]. In this paper, the simulation is extended by network virtualization. Diﬀerent approaches are described and evaluated. The remainder of this paper is structured as follows: In Section 2 the related work is discussed. Section 3 introduces the energy-eﬃcient VHE architecture. In Section 4 virtualization methods are discussed in detail, which can be used to realize the suggested VHE architecture. Section 5 describes the overlays that are implemented in the simulation of VHE and evaluates their beneﬁts and overheads. Section 6 concludes this paper.

2

Related Work

The 3rd Generation Partnership Project (3GPP) [7] describes a VHE as a concept for personal service environment portability across network boundaries and between terminals. Users are consistently presented with the same personalized features, interface customizations and services in whatever network and whatever terminal, whereever the user may be located. Further the Open Service Access (OSA) framework for separating network and service layers was proposed by

Network Virtualization in Future Home Environments

179

3GPP. The OSA speciﬁed in conjunction with the Parlay Application Programming Interfaces the base technology that was applied in the project VESPER [8,9]. The project aimed to deﬁne, demonstrate, and promote a service architecture for provision of VHE across a multi-provider, heterogeneous network and system infrastructure. The European Institute for Research and Strategic Studies in Telecommunications1 (Eurescom) [10] described a VHE as an environment enabling users to receive customized and personalized services, regardless of location, access network or terminal type in a way that users will not see a diﬀerence in using services at home or while roaming in other networks. Similarly for Liotta et al. [11] the VHE concept pursues the idea of service universality, which allows users to transparently access services anytime, anywhere, with any type of terminal. This concept allows users to be consistently presented with the same personalized features and preferences, regardless of the context. Nakajima et al. [12] proposed a virtual overlay network for integrating networked home appliances while also considering media streaming and disk sharing. All of this work done in the ﬁeld of VHE assumes external providers for operating services inside a home. In contrast, the VHE proposed in this paper realizes a home centric view on the network, where virtualization is the clue to aggregate and consolidate distributed hardware resources. This understanding of a virtualization based VHE architecture was already introduced in [3] and deepened with more simulation results in [5]. An economic model for fostering fair resource sharing was presented in [4]. The investigations were extended in [6] by the creation of a prototype for task virtualization for sending virtual machines between homes designed for minimal resource usage. Future Internet Platforms like PlanetLab [13] envision open platforms for distributed end-to-end applications, similar to VHE. Network resources and hardware resources of end-hosts that are located all over the world are virtualized. A user is provided a slice that consists of hundreds of shells, one for each end-host. Comparable approaches towards end-to-end virtualization are done by other projects (e.g. GENI2 or VINI [14]). These kinds of network virtualization are comparable to the VHE architecture with respect to virtualization. However, the focus of these approaches is on providing Future Internet environments, rather than sharing hardware resources among end users.

3

The VHE Architecture

The VHE architecture approach as it was introduced in [3,4,5,6] enables the energy-eﬃcient sharing of hardware resources amongst home networks. The main goal of the architecture is to achieve a consolidation of load (e.g. in terms of bandwidth consumption, CPU usage or disk space). Especially the load generated by always-on applications is considered. The number of applications that requires always-on hardware (e.g. media-server or ﬁle-sharing client) in home networks 1 2

http://www.eurescom.de http://www.geni.net

180

A. Berl et al.

Fig. 1. VHE architecture

is growing fast, leading to a high number of computers running on a 24/7 basis. Similarly to approaches in data centers, the overall load is shifted to a small number of computers, in order to relieve others. Unloaded computers can be hibernated (or turned oﬀ) to save energy. To achieve this, a possibly large number of home networks is interconnected to share their resources. Consolidation of load is envisioned as follows: A user starts a task (e.g. a ﬁle-sharing client) locally on his computer. The VHE environment discovers potential of energy savings. It moves the task to another computer (probably in another home network) that is already running and has enough hardware resources left to process the task. The local computer can be turned oﬀ to save energy. When the task is ﬁnished, the local computer is turned on and the result is sent back. In this way only a small portion always-on computers is needed. A home network usually consists of a small number of computers, e.g. PCs, home servers, or laptops and a gateway to the Internet. In the VHE architecture, a gateway is a Linux-based diskless always-on computer that has small energy needs (e.g. an AVM FritzBox3 ). The gateway maintains a permanent entry to the interconnected home networks and represents its home network in the VHE. The interconnected home networks form a distributed pool of virtual resources, and the architecture uses a distributed management to allocate resources to home networks dynamically. To allow energy saving, a distinction is made between active and passive home networks (contributing and not contributing home networks). A home network is called active if it contains at least one computer which is turned on. In a passive home network only the gateway is online and other hardware is hibernated. The VHE architecture is illustrated in Figure 1. In this example four home networks are interconnected, two active and two passive homes. In the ﬁgure load is migrated from an end-host in the active home network b to an end-host in the active home network c. The end-host in home network b can be hibernated or turned oﬀ after the migration process. If no further computer is turned on in home network b, it can change its status to passive. 3

http://www.avm.de/en/Produkte/FRITZBox/index.html

Network Virtualization in Future Home Environments

4

181

Virtualization Methods in the VHE Architecture

VHE combines two diﬀerent virtualization approaches to enable an energyeﬃcient resource sharing among home networks, virtualization of host resources and network virtualization. 4.1

Virtualization of Host Resources

To enable hardware resource sharing (e.g. CPU cycles, bandwidth or disk space) among end-hosts in home networks, it is necessary to make idle resources available for processes (guest applications) of other end-hosts (resource availability, see Section 1). A runtime environment is needed that processes guest applications of other users. This runtime environment has to be ﬂexible enough to enable the processing of a wide variety of guest applications. The guest application might come from a diﬀerent Operating Systems (OS) (e.g. Windows, MAC, or Linux) or from a diﬀerent computer architecture (e.g. x86 or PowerPC). This ﬂexible runtime environment has also to deal with privacy and security issues. On the one hand, guest applications are sent to unknown hosts within the VHE architecture. The guest applications might come together with private data (e.g. a movie that needs to be encoded). The owner of the guest application wants it to be separated as much as possible from the user environment of the host. On the other hand, a user that hosts a guest application wants his machine to be separated as much as possible from the guest application. Nowadays, the approach of system virtualization is successfully used to consolidate services in data centers. Several services can run separately on top of a single hardware, saving hardware costs, space, and energy. In [6] system virtualization based on QEMU [15] has been used as initial VHE solution to virtualize idle resources of hosts. In system virtualization, a Virtual Machine (VM) is created, i.e. a full machine is virtualized, consisting of virtual CPUs, virtual memory, virtual hard disk, virtual Network Interface Card, etc. A VM is a perfect recreation of a real machine in such a way that an OS can be installed on it without being aware of the resource virtualization. Typical examples for system virtualization software are, e.g., XEN [16] or VMWare ESX Server4 ). In addition to providing a VM, QEMU emulates the CPU within the VM, based on dynamic binary translation [15]. This kind of emulation enables QEMU based VMs to migrate between diﬀerent architectures (e.g. x86 or Mac) and makes QEMU a very ﬂexible choice for VHE. First results with QEMU-based virtualization are described in [6]. 4.2

Network Virtualization

Home networks that are a part of the VHE architecture have to be interconnected. Participating homes have to be addressable for other homes in order 4

http://www.vmware.com/products/vi/esx

182

A. Berl et al.

to enable communication within the connected VHE (home network interconnection, see Section 1). In the VHE architecture no central VHE provider is intended. Therefore, the addressing of the home networks has to be solved in a distributed way to make participating home networks accessible. Furthermore, QEMU-based virtualization makes idle resources on hosts available in separated runtime environments, however they are not yet accessible for other users. A mediation of available hardware resources has to established (resource mediation, see Section 1). Idle resources have to be discovered within the VHE network and they have to be made addressable to enable the allocation of idle resources to other participants. Another important requirement of the VHE architecture is the distributed management of resources (resource allocation, see Section 1). No central architectural element is available in VHE that manages the balanced cooperation of home networks and the access to available resources. This cooperation has to be achieved in a distributed way. Energy-eﬃcient resource sharing has a number of constraints that have to be considered. Examples are fair distribution of energy consumption or the provision of a suﬃcient quality of service to users. A cost model to target such constraints in VHE is discussed in [4]. The distributed management has to be aware of the diﬀerent states of the home networks (active, passive) and the resources that are available at a certain point of time. In addition, guest applications might have special needs in terms of hardware resources that have to be considered when resources are allocated. Such requirements of the VHE architecture can be met by network virtualization methods [17,18]. Two kinds of virtualized networks are widely used today: Virtual Local Area Networks (VLANs) and Virtual Private Networks (VPNs). VLANs like IEEE 802.1Q5 operate mainly on the link layer, subdividing a switched Local Area Network into several distinct groups either by assigning the diﬀerent ports of a switch to diﬀerent VLANs or by tagging link layer frames with VLAN identiﬁers and then routing accordingly. VPNs like IPSec6 , on the other hand, establish a network layer tunnel to either connect two networks (site-to-site), one network and a host (site-to-end ) or two hosts (end-to-end ) with an encrypted and/or authenticated channel over the Internet. However, these kinds of virtualization methods target mainly the sharing of links among users and are not suﬃcient for the VHE approach. Besides the virtualization of links, also the virtualization of routers has been investigated in several approaches. In [19] system virtualization (e.g. based on XEN [16] or VMWare ESX Server)) is applied to routers to create virtualized networks with special features. In [20,21] performance challenges are identiﬁed that have to be tackled when virtual routers are based on XEN. Other forms of router virtualization are already available in commercial products7 . However, virtualization of routers does not solve the network virtualization issues of the VHE architecture. Such solutions mainly allow the concurrent usage of network 5 6 7

http://standards.ieee.org/getieee802/download/802.1Q-2003.pdf http://tools.ietf.org/html/rfc4301 http://www.cisco.com/en/US/docs/ios xr sw/iosxr r3.2/interfaces/command /reference/hr32lr.html

Network Virtualization in Future Home Environments

183

infrastructure. In the VHE, a mediation of available hardware resources and their distributed management is needed within the VHE. A further approach towards network virtualization are peer-to-peer (P2P) overlays [22]. In this approach logical links are deﬁned on top of a physical infrastructure. A single logical hop in the overlay can be mapped to several physical hops in the network. P2P networks are mainly classiﬁed according to their architecture and their algorithmic features. In pure P2P overlays (e.g. Chord [23]) all peers are assumed to be equal. In hybrid P2P overlays some peers are distinguished from other peers, i.e. some peers have diﬀerent capabilities than others (e.g. eDonkey [24]). P2P overlays are denoted to be unstructured if the algorithms establish overlay links which do not follow a regular connectivity pattern. In contrast, P2P overlays are said to be structured if a generic but predeﬁned organization scheme (e.g. a ring) of the overlay exists. In contrast to the previously mentioned network virtualization approaches, P2P overlays do not only virtualize links and nodes, but they solve three main VHE issues: Home network interconnection, resource mediation, and resource allocation. P2P networks establish an addressing scheme within the overlay that enables the addressing of peers as well as the addressing of available resources. In P2P ﬁle-sharing networks (e.g. eDonkey), ﬁles can be discovered and addressed, whereas in P2P VoIP applications like Skype8 users are locatable and addressable. This solves the problem of home network interconnection as well as the resource mediation problem. Also a management with respect to resource allocation is provided in such P2P overlays. eDonkey, e.g. establishes a complex tit-for-tat principle to enable a fair resource sharing and allows the distributed download of ﬁles from diﬀerent sources concurrently. There are several P2P overlays available that can be used within the VHE architecture. Solutions have to be scalable (with a high number of home networks) and they need to be lightweight to operate on the always-on gateways (see Section 3). Pastry [25], a structured pure P2P overlay, has been used as initial network virtualization technology in [6]. Pastry is scalable and available as open source platform. Although Pastry was successfully used in simpliﬁed initial tests, it has its shortcomings. First, structured overlays like Pastry are more vulnerable to high churn-rates9 than unstructured networks [22]. However, end-hosts might show a very dynamic behavior in the VHE architecture, concerning on-line and oﬀ-line times. Another problem is that pastry only solve the home network interconnection and the resource mediation problem, resource allocation (as described above) is not addressed in Pastry. In this paper the unstructured and hybrid eDonkey P2P overlay is suggested as a solution for the VHE network virtualization. It has the capability to solve all of the mentioned requirements and is resistant to high churn rates. eDonkey is a very popular ﬁle sharing P2P network with a high amount of users (and trafﬁc) [24] that has practically proven to be very scalable. Another reason for this choice is the similarity of the hybrid eDonkey structure to the VHE structure 8 9

http://www.skype.com In P2P parlance, the term churn denotes the stochastic process of peer turnover as occurring when peers join or leave the system.

184

A. Berl et al.

(end-hosts and gateways). Two kinds of nodes are participating in the eDonkey network: peers and super nodes (index servers). Peers are providing and consuming resources, similar to the end-hosts in the VHE architecture. Super nodes form a separate overlay to share information. A peer can report its available resources to the super node and request the location of hardware resources from the super node. The location of a gateway within the VHE architecture is very similar to the location of a super node — the gateway is physically the ﬁrst node of each home network. This location makes the gateway the natural place for gathering statistics about the home network that are needed to enable a fair and energy-eﬃcient resource sharing among home networks. In addition, the gateway is supposed to be always-on, which enables it to manage and distribute information among other gateways.

5

Evaluation of Network Virtualization Overhead

Simulation results concerning VHE have already been shown in [3,4,5,6]. The simulation itself is a discrete event simulation built with the general purpose programming language Java. Simulated is a meshed network of homes with certain resources (CPU time, disk space, uplink and downlink bandwidth). A home consists of at least one computer and a gateway. Intra- and interhome communication underlies certain delays and links certain latencies. Homes oﬀer an amount of shareable resources. Load is modeled as tasks that are executed on homes and migrated between them. Homes cycle through states that indicate times of work or rest. Further details about initial parameters of simulation runs can be taken from previous works. In this paper, the simulation is extended by network virtualization approaches. We refer to the application “Download Sharing” (DS) introduced in previous work; a network of interconnected homes exchange (share) download tasks (DStasks) for aggregating load on a part of the network with the aim of power saving. In this work we use DS to investigate the traﬃc produced by our approach under diﬀerent organization schemes. We do not address power saving at all, but want to know how much traﬃc originate on those nodes which maintain statistics. A DS-task is a description of desired content (music, video, etc.) and maximum allowed download bandwidth allocatable for its download. Homes play two roles; as initiator they send out DS-tasks to other homes and receive results later or as executer they receive DS-tasks, download the desired content, and then upload the completed DS-task back to initiators. There is a communication protocol between homes that can be divided into a state phase and a resource phase; in the state phase state information about homes is exchanged, whereas during the resource phase available resources are located and allocated. State information about homes includes the current state of the home (see Figure 1) and the amount of free resources available for executing DS-tasks. Conceptually the DS application is based on virtual machines (VMs) encapsulating all necessary parts for migrating a DS-task before and after execution. These VMs are rather small on initiator side and grow to considerable ﬁle sizes

Network Virtualization in Future Home Environments

Fig. 2. Topology and information ﬂow Overlay 1

185

Fig. 3. Topology and information ﬂow Overlay 2

on executer side because they including now the download content. Therefore, a constraint is the possible usable bandwidth between initiator and executer. Only for investigating the signaling traﬃc a server-based approach is compared to two unstructured hybrid overlays in terms of traﬃc overhead, caused by diﬀerent strategies of information management. Initially, a simpliﬁed centralized Server -based approach has been simulated in VHE. Each Home network (H) sends state information to one single H that acts as Super Home (SH), which has a global view on the system. In case of task-migrations, all resource requests must be sent to that SH. The second approach (called Overlay 1 ) shown in Figure 2, is based on an unstructured and hybrid overlay (inspired by eDonkey). A number of SHs are deﬁned, which cluster the network. A SH is a H that additionally acts as a server for a cluster of Hs (clusters are illustrated as squares in Figure 2). Every state change information within a cluster is replicated at all other SHs (SHs have a global view). Resource requests of Hs are answered by their corresponding SH. The arrows in Figure 2 illustrate two independent communication ﬂows. With message S1 a H sends its state to the SH. The SH forwards the state message to all other SHs (S2). At this point, the state information is replicated amongst SHs. Each state change triggers a state message, therefore a considerable amount of messages is generated. To gain resources, a resource request (R1) is sent to to a corresponding SH. The SH replies with a list of currently active Hs (R2). Both of these approaches, Server and Overlay 1 create a global view on the VHE, causing overhead. However, the VHE resource management is mainly done by lightweight always-on gateways as described in Section 3 and the overhead needs to be reduced. To achieve this, a third approach, Overlay 2 is suggested (again eDonkey inspired) that has modiﬁed communication patterns (shown in Figure 3). It has the aim of keeping resource information as local as possible. Hs send state information to their corresponding SH (message S1). SHs do not replicate this information, but exchange meta information about free hardware resources within the own cluster in conﬁgurable time-intervals (S2). In contrast

186

A. Berl et al. Table 1. Message sizes in byte

Message type Overlay 1 Overlay 2 State update S (S1, S2) S (S1, S2) Resource request R (R1) R (R1, R2b) Resource response L × S (R2) R (R2a, R3)

to Overlay 1 none of the SHs has a global view on available resources. Resource information is kept local within each SH’s cluster, only meta information is exchanged. This results in a diﬀerent resource request scheme. The resource request ﬁrst goes to the responsible SH (R1). The SH checks its own cluster and forwards the request to a H with free resources if possible (R2); this H directly answers the requester (R3). Otherwise, if there is no host within the own cluster that can process the task, the SH contacts other SHs, based on the available meta information. If another SH has enough idle resources in its cluster, the initial SH forwards the resource request. Costs of resource management in the system investigated is mainly based on the number and size of signaling messages within the network. To understand the message complexity of the presented overlays the sizes of modeled messages according to the introduced communication protocol are explained. The message sizes in byte are speciﬁed in Table 1. S = 60 byte is the size of a state message, R = 24 byte is the size of a resource message, and L is the number of entries in the list of active Hs within a SH’s cluster in Overlay 1. State update messages in byte/s for Server can be approximated by

for Overlay 1 by

N M S(T + E) , Y

(1)

N M S(T + E) C × 2− , Y N

(2)

and for Overlay 2 by Y UH

C + 2 UYSH ( N C − 1) Y

S

(3)

where N is the number of Hs (network size), M is the assumed load (number of DS-tasks per H per year), T is the number of state transitions per H according to Figure 1 if homes staying in state active for executing tasks or staying in state passive for saving power, E is the number of state events per task as deﬁned in the communication protocol of Overlay 1, Y is the simulation time in seconds (one year), and C is the cluster size (each SH manages C Hs). UH & USH indicate the delay between state updates from Hs and SHs as deﬁned in the communication protocol of Overlay 2.

Network Virtualization in Future Home Environments

187

Accordingly, the traﬃc of resource messages in byte/s per SH for Server can be approximately calculated by

for Overlay 1 by

M (R + LS) N, Y

(4)

M (R + LS) C, Y

(5)

and for Overlay 2 by 2(Pa M C + Pb M C( N C − 1)) R (6) Y which shows that the critical parameter N dramatically increases the message complexity for Server, whereas Overlay 1 only relies on the number of clusters C which is a predeﬁned fraction of N . For Overlay 2 additional parameters Pa & Pb are necessary. Those parameters express the probability if a task can be processed within the requester’s cluster (Pa ) or if it is forwarded to another cluster (Pb ). Table 2. Expected state update traﬃc per SH in byte/s

N Server Overlay1 100 0.69 1.21 1000 6.93 13.68 10000 69.25 138.34 100000 692.54 1384.91

Overlay2 1.65 6.45 54.45 534.45

Table 2 presents state update traﬃc in a network with C = 25 with N increasing. The number of tasks per week per home (λ) is ﬁxed to λ = 10, one year is considered, UH is 1200 s and USH is 900 s. As the table clearly shows, traﬃc is correlated with network size N . For veriﬁcation and comparison all overlays are simulated (in a discrete event simulation [3,4,5,6]). The results are illustrated in Figures 4 and 5. Figure 4 shows state and resource traﬃc in byte/s with regard to N . Generally it can be seen that state traﬃc is much more critical than resource traﬃc. For state traﬃc Server and Overlay 2 clearly outperform Overlay 1. For N > 700 Server is also outperformed by Overlay 2 ; however state traﬃc is still linear and a matter of scalability. Further the resource traﬃc of Server grows with N . Figure 5 shows the same for ﬁxed N and increasing λ. Again, the state traﬃc is the more critical one. Overlay 2 clearly outperforms Overlay 1 for λ > 6 and Server for λ > 12. Overall, Overlay 2 exhibits the good property of being invariant to load. Even the overhead caused by Overlay 1 does not constrict Hs with high synchronous access bandwidth like 50 Mbit/s to act as SH. After this worst case analysis about the feasibility of our approach the next step will be to choose a P2P-system which best ﬁts to our approach of energy eﬃcient resource sharing for homes but also minimizes communication overhead.

188

A. Berl et al.

Traffic per super home 14 12

byte / s

10 8

State Server State Overlay 1 State Overlay 2 Resource Server Resource Overlay 1 Resource Overlay 2

6 4 2 0 100

200

300

400

500

600

700

800

900

1000

Number of homes (N) Fig. 4. Traﬃc in terms of network size with a ﬁxed load of 10 tasks per week

Traffic per super home 20 18 16

byte / s

14 12

State Server State Overlay 1 State Overlay 2 Resource Server Resource Overlay 1 Resource Overlay 2

10 8 6 4 2 0 5

10

15

20

25

30

35

Tasks per week per home (λ) Fig. 5. Traﬃc in terms of load with a ﬁxed network size of 400 Hs

6

Conclusion and Future Work

This paper has presented an energy-eﬃcient virtual home environment that enables resource sharing among users in home networks. Particularly, the virtualization-related aspects of the suggested environment were discussed. Therefore,

Network Virtualization in Future Home Environments

189

virtualization-related requirements of a future home environment were deﬁned and virtualization methods were selected to meet these requirements. In this context, an overview on network virtualization approaches was given to motivate the selection of virtualization solutions. Simulations that are described in previous papers [3,4,5,6] have been extended by simpliﬁed network virtualization approaches. In this paper these approaches were evaluated and compared to each other concerning their eﬀects on the energy-eﬃcient operation of the envisioned future home environment. The overhead which is imposed by the network virtualization approaches was illustrated and discussed. The paper showed that the chosen instance of network virtualization is conceptually appropriate for the future home environment architecture. However, it has to be improved in terms of overhead and scalability in future work. Additionally, it has to be evaluated in future works, in which way the distributed ﬁle-sharing management of the suggested network virtualization approach can be adapted to enable a balanced and energy-eﬃcient resource-sharing management in future home environments.

References 1. Koomey, J.G.: Estimating total power consumption by servers in the US and the world. Final report. 15 (February 2007) 2. Bertoldi, P., Atanasiu, B.: Electricity consumption and eﬃciency trends in the enlarged European Union. IES–JRC. European Union (2007) 3. Hlavacs, H., Hummel, K.A., Weidlich, R., Houyou, A., Berl, A., de Meer, H.: Energy Eﬃciency in Future Home Environments: A Distributed Approach. In: IFIP International Federation for Information Processing: Home Networking, Proceedings of First IFIP WG 6.2 Home Networking Conference (IHN 2007), Paris, France, December 10-12. Springer, Berlin (2007) 4. Garc´ıa, A.E., Berl, A., Hummel, K.A., Weidlich, R., Houyou, A., Hackbarth, K.D., de Meer, H., Hlavacs, H.: An Economical Cost Model for fair resource sharing in Virtual Home Environments. In: Proceedings of Next Generation Internet Networks (NGI 2008), Krakow, Poland, April 28-30, pp. 153–160. IEEE Computer Society Press, Los Alamitos (2008) 5. Hlavacs, H., Hummel, K.A., Weidlich, R., Houyou, A., Berl, A., de Meer, H.: Distributed Energy Eﬃciency in Future Home Environments. Annals of Telecommunication: Next Generation Network and Service Management 63(9-10), 473–485 (2008) 6. Hlavacs, H., Weidlich, R., Treutner, T.: Energy Saving in Future Home Environments. In: 2nd Home Networking Conference at IFIP Wireless Days, Dubai, United Arab Emirates (November 2008) 7. Pope, M., Meredith, J.: The Virtual Home Environment, Release 5. TR 22.121 V5.3.1, 3GPP (July 2002) 8. Roque, R., Soares, T., Oliveira, J.: VESPER Project- Validation of VHE Concept (2001) 9. Roussaki, I., Jormakka, H., Xynogalas, S., Laikari, A., Chantzara, M., Anagnostou, M.: Multi-terminal and multi-network access to virtual home environment. In: IST Mobile and Wireless Telecommunications, Thessaloniki, Greece (June 2002)

190

A. Berl et al.

10. Geuna: UMTS Network Aspects. EURESCOM Project P920, VHE Trial review report, Deliverable 4 (Januvary 2001) 11. Liotta, A., Yew, A., Bohoris, C., Pavlou, G.: Supporting adaptation-aware services through the virtual home environment. In: Proc. HP-OVUA, Hewlett-Packard Openview University Association Plenary Workshop, Citeseer, pp. 11–13 (2002) 12. Nakajima, T., Ueno, D., Tokunaga, E., Ishikawa, H., Satoh, I., Aizu, H.: A Virtual Overlay Network for Integrating Home Appliances. In: 2002 Symposium on Applications and the Internet (SAINT 2002), Nara, Japan (July 2002) 13. Chun, B., Culler, D., Roscoe, T., Bavier, A., Peterson, L., Wawrzoniak, M., Bowman, M.: Planetlab: an overlay testbed for broad-coverage services. SIGCOMM Comput. Commun. Rev. 33(3), 3–12 (2003) 14. Bavier, A., Feamster, N., Huang, M., Peterson, L., Rexford, J.: In vini veritas: realistic and controlled network experimentation. In: SIGCOMM 2006: Proceedings of the 2006 conference on Applications, technologies, architectures, and protocols for computer communications, pp. 3–14. ACM, New York (2006) 15. Bellard, F.: QEMU, a fast and portable dynamic translator. In: Proceedings of the USENIX Annual Technical Conference, FREENIX Track, pp. 41–46 (2005) 16. Barham, P., Dragovic, B., Fraser, K., Hand, S., Harris, T., Ho, A., Neugebauer, R., Pratt, I., Warﬁeld, A.: Xen and the art of virtualization. SIGOPS Oper. Syst. Rev. 37(5), 164–177 (2003) 17. Chowdhury, N.M.M.K., Boutaba, R.: A Survey of Network Virtualization. University of Waterloo (October 2008), http://www.cs.uswaterloo.ca/research/tr/2008/CS-2008-25.pdf 18. Feamster, N., Gao, L., Rexford, J.: How to lease the internet in your spare time. SIGCOMM Comput. Commun. Rev. 37(1), 61–64 (2007) 19. Berl, A., Fischer, A., de Meer, H.: Using System Virtualization to Create Virtualized Networks. In: Workshops der Wissenschaftlichen Konferenz Kommunikation in Verteilten Systemen (WowKiVS2009), Kassel, Germany, March 2-6 (2009); Electronic Communications of the EASST, vol. 17, EASST (March 2009) 20. Egi, N., Greenhalgh, A., Handley, M., Hoerdt, M., Mathy, L., Schooley, T.: Evaluating XEN for Router Virtualization. In: 16th Int. Conf. on Comp. Commun. and Networks - ICCCN 2007, pp. 1256–1261 (August 2007) 21. Menon, A., Cox, A.L., Zwaenepoel, W.: Optimizing Network Virtualization in XEN. In: USENIX Annual Technical Conference, pp. 15–28 (May 2006) 22. Steinmetz, R., Wehrle, K.: What is this Peer-to-peer about? In: Peer-to-Peer Systems and Applications. LNCS, vol. 3485, pp. 9–16. Springer, Heidelberg (2005) 23. Stoica, I., Morris, R., Karger, D., Kaashoek, M.F., Balakrishnan, H.: Chord: A scalable peer-to-peer lookup service for internet applications. In: SIGCOMM 2001: Proceedings of the 2001 conference on Applications, technologies, architectures, and protocols for computer communications, pp. 149–160. ACM Press, New York (2001) 24. Tutschku, K.: A measurement-based traﬃc proﬁle of the eDonkey ﬁlesharing service. In: Barakat, C., Pratt, I. (eds.) PAM 2004. LNCS, vol. 3015, pp. 12–21. Springer, Heidelberg (2004) 25. Rowstron, A., Druschel, P.: Pastry: Scalable, distributed object location and routing for large-scale peer-to-peer systems. In: IFIP/ACM Int. Conference on Distributed Systems Platforms (Middleware), Heidelberg, vol. 11, pp. 329–350 (2001)

Author Index

Araujo Wickboldt, Juliano 71, 138 Armando Bianchin, Lu´ıs 71 Bartolini, Claudio 71, 138 Berl, Andreas 177 Bonato Both, Cristiano 71, 138 Campos Nobre, J´eferson 99 Castagna Lunardi, Roben 71, 138 Christodoulou, Athena 152 Clematis, A. 43 Corana, A. 43 D’Agostino, D. 43 da Costa Cordeiro, Weverton Luis 71, 138 de Boer, Pieter-Tjerk 164 de Meer, Hermann 177 Diao, Yixin 123 Duarte Jr., Elias P. 29 Edwards, Nigel

56

Galizia, A. 43 Girardi Andreis, Fabr´ıcio Graﬀ, Daniel 111

71, 138

Parzyjegla, Helge 111 Paschoal Gaspary, Luciano 71, 138 Pras, Aiko 164 Quarati, A.

43

Racz, Peter 1 Richling, Jan 111 Rolia, Jerry 56 Sadre, Ramin 164 Sch¨ onw¨ alder, J¨ urgen 15 Schrank, Michael 177 Schr¨ oter, Arnd 111 Sperotto, Anna 164 Stiller, Burkhard 1 Trastour, David

71, 138, 152

Weidlich, Roman 177 Wilcock, Lawrence 56

Hagen, Sebastian 56 Heching, Aliza 123 Hlavacs, Helmut 177 Kiefer, Andreas 29 Kirschnick, Johannes 56 Konstantinou, Alexander V.

Marinov, Vladislav 15 Morariu, Cristian 1 M¨ uhl, Gero 111 Murta, Cristina D. 29

Yemini, Yechiam

85

85

Zambenedetti Granville, Lisandro 99, 138

71,

IEEE International Workshop on Distributed Systems: Operations and Management,

IEEE International Workshop on Distributed Systems: Operations and Management, DSOM 2005, Barcelona, Spain, October 24-26,

Wired-Wireless Multimedia Networks and Services Management: 12th IFIP IEEE International Conference on Management of Multimedia and Mobile Networks and ... Networks and Telecommunications)

Wired-Wireless Multimedia Networks and Services Management: 12th IFIP IEEE International Conference on Management of Multimedia and Mobile Networks ... Networks and Telecommunications)

Knowledge management: systems and processes

Mechanisms for Autonomous Management of Networks and Services: 4th International Conference on Autonomous Infrastructure, Management, and Security, ... Networks and Telecommunications)

Utility Computing: 15th IFIP IEEE International Workshop on Distributed Systems: Operations and Management, DSOM 2004, Davis, CA, USA, November 15-17, ... (Lecture Notes in Computer Science)

Management of Telecommunication Systems and Services

Utility Computing: 15th IFIP IEEE International Workshop on Distributed Systems: Operations and Management, DSOM 2004, Davis, CA, USA, November 15-17, ... (Lecture Notes in Computer Science)

Management of Multimedia Networks and Services: 8th International Conference on Management of Multimedia Networks and Services, MMNS 2005, Barcelona,

Parallel and Distributed Simulation Systems

Operations and Management in IP-Based Networks: 5th IEEE International Workshop on IP Operations and Management, IPOM 2005, Barcelona, Spain, October

Testing of Software and Communication Systems: 21st IFIP WG 6.1 International Conference, TESTCOM 2009 and 9th International Workshop, FATES 2009, ... Networks and Telecommunications)

Testing of Software and Communication Systems: 21st IFIP WG 6.1 International Conference, TESTCOM 2009 and 9th International Workshop, FATES 2009, Eindhoven, ... Networks and Telecommunications)

Integrated Management of Systems, Services, Processes and People in IT: 20th IFIP IEEE International Workshop on Distributed Systems: Operations and ... Networks and Telecommunications)

IEEE International Workshop on Distributed Systems: Operations and Management,

IEEE International Workshop on Distributed Systems: Operations and Management,

IEEE International Workshop on Distributed Systems: Operations and Management, DSOM 2005, Barcelona, Spain, October 24-26,

Wired-Wireless Multimedia Networks and Services Management: 12th IFIP IEEE International Conference on Management of Multimedia and Mobile Networks and ... Networks and Telecommunications)

Wired-Wireless Multimedia Networks and Services Management: 12th IFIP IEEE International Conference on Management of Multimedia and Mobile Networks ... Networks and Telecommunications)

Knowledge management: systems and processes

Knowledge management: systems and processes

Next Generation Telecommunications Networks, Services, and Management (IEEE Press Series on Network Management)

Next Generation Telecommunications Networks, Services, and Management (IEEE Press Series on Network Management)

Distributed Systems Security: Issues, Processes and Solutions

Mechanisms for Autonomous Management of Networks and Services: 4th International Conference on Autonomous Infrastructure, Management, and Security, ... Networks and Telecommunications)

Utility Computing: 15th IFIP IEEE International Workshop on Distributed Systems: Operations and Management, DSOM 2004, Davis, CA, USA, November 15-17, ... (Lecture Notes in Computer Science)

Management of Telecommunication Systems and Services

Utility Computing: 15th IFIP IEEE International Workshop on Distributed Systems: Operations and Management, DSOM 2004, Davis, CA, USA, November 15-17, ... (Lecture Notes in Computer Science)

Atmospheric Processes and Systems

Dynamical systems and processes

Telecommunications on Systems Engineering

1996 IEEE International Conference on Systems, Man and Cybernetics

Management of Multimedia Networks and Services: 8th International Conference on Management of Multimedia Networks and Services, MMNS 2005, Barcelona,

Parallel and Distributed Simulation Systems

Distributed Systems: Principles and Paradigms

Distributed Systems: Concepts and Design

Distributed Systems: Principles and Paradigms

Distributed Processing and Database Systems

Operations and Management in IP-Based Networks: 5th IEEE International Workshop on IP Operations and Management, IPOM 2005, Barcelona, Spain, October

Testing of Software and Communication Systems: 21st IFIP WG 6.1 International Conference, TESTCOM 2009 and 9th International Workshop, FATES 2009, ... Networks and Telecommunications)

Testing of Software and Communication Systems: 21st IFIP WG 6.1 International Conference, TESTCOM 2009 and 9th International Workshop, FATES 2009, Eindhoven, ... Networks and Telecommunications)

Interactive Distributed Multimedia Systems and Telecommunication Services: 7th International Workshop, IDMS 2000 Enschede, The Netherlands, October

Cooperative Management of Enterprise Networks (Network and Systems Management)

Cooperative Management of Enterprise Networks (Network and Systems Management)

Integrated Management of Systems, Services, Processes and People in IT: 20th IFIP IEEE International Workshop on Distributed Systems: Operations and ... Networks and Telecommunications)

IEEE International Workshop on Distributed Systems: Operations and Management,

IEEE International Workshop on Distributed Systems: Operations and Management,

IEEE International Workshop on Distributed Systems: Operations and Management, DSOM 2005, Barcelona, Spain, October 24-26,

Wired-Wireless Multimedia Networks and Services Management: 12th IFIP IEEE International Conference on Management of Multimedia and Mobile Networks and ... Networks and Telecommunications)

Wired-Wireless Multimedia Networks and Services Management: 12th IFIP IEEE International Conference on Management of Multimedia and Mobile Networks ... Networks and Telecommunications)

Knowledge management: systems and processes

Knowledge management: systems and processes

Next Generation Telecommunications Networks, Services, and Management (IEEE Press Series on Network Management)

Next Generation Telecommunications Networks, Services, and Management (IEEE Press Series on Network Management)

Distributed Systems Security: Issues, Processes and Solutions

Mechanisms for Autonomous Management of Networks and Services: 4th International Conference on Autonomous Infrastructure, Management, and Security, ... Networks and Telecommunications)

Utility Computing: 15th IFIP IEEE International Workshop on Distributed Systems: Operations and Management, DSOM 2004, Davis, CA, USA, November 15-17, ... (Lecture Notes in Computer Science)

Management of Telecommunication Systems and Services

Utility Computing: 15th IFIP IEEE International Workshop on Distributed Systems: Operations and Management, DSOM 2004, Davis, CA, USA, November 15-17, ... (Lecture Notes in Computer Science)

Atmospheric Processes and Systems

Dynamical systems and processes

Telecommunications on Systems Engineering

1996 IEEE International Conference on Systems, Man and Cybernetics

Management of Multimedia Networks and Services: 8th International Conference on Management of Multimedia Networks and Services, MMNS 2005, Barcelona,

Parallel and Distributed Simulation Systems

Distributed Systems: Principles and Paradigms

Distributed Systems: Concepts and Design

Distributed Systems: Principles and Paradigms

Distributed Processing and Database Systems

Operations and Management in IP-Based Networks: 5th IEEE International Workshop on IP Operations and Management, IPOM 2005, Barcelona, Spain, October

Testing of Software and Communication Systems: 21st IFIP WG 6.1 International Conference, TESTCOM 2009 and 9th International Workshop, FATES 2009, ... Networks and Telecommunications)

Testing of Software and Communication Systems: 21st IFIP WG 6.1 International Conference, TESTCOM 2009 and 9th International Workshop, FATES 2009, Eindhoven, ... Networks and Telecommunications)

Interactive Distributed Multimedia Systems and Telecommunication Services: 7th International Workshop, IDMS 2000 Enschede, The Netherlands, October

Cooperative Management of Enterprise Networks (Network and Systems Management)

Cooperative Management of Enterprise Networks (Network and Systems Management)

Recommend Documents