Innovative Applications in Data Mining

Nadia Nedjah, Luiza de Macedo Mourelle, and Janusz Kacprzyk (Eds.) Innovative Applications in Data Mining Studies in C...

Author: Nadia Nedjah | Nadia Nedjah | Luiza de Macedo Mourelle | Janusz Kacprzyk

60 downloads 1881 Views 23MB Size Report

This content was uploaded by our users and we assume good faith they have the permission to share this book. If you own the copyright to this book and it is wrongfully on our website, we offer a simple DMCA procedure to remove your content from our site. Start by pressing the button below!

Report copyright / DMCA form

DOWNLOAD PDF

Nadia Nedjah, Luiza de Macedo Mourelle, and Janusz Kacprzyk (Eds.) Innovative Applications in Data Mining

Studies in Computational Intelligence, Volume 169 Editor-in-Chief Prof. Janusz Kacprzyk Systems Research Institute Polish Academy of Sciences ul. Newelska 6 01-447 Warsaw Poland E-mail: [email protected] Further volumes of this series can be found on our homepage: springer.com Vol. 148. Philipp Limbourg Dependability Modelling under Uncertainty, 2008 ISBN 978-3-540-69286-7 Vol. 149. Roger Lee (Ed.) Software Engineering, Artificial Intelligence, Networking and Parallel/Distributed Computing, 2008 ISBN 978-3-540-70559-8 Vol. 150. Roger Lee (Ed.) Software Engineering Research, Management and Applications, 2008 ISBN 978-3-540-70774-5 Vol. 151. Tomasz G. Smolinski, Mariofanna G. Milanova and Aboul-Ella Hassanien (Eds.) Computational Intelligence in Biomedicine and Bioinformatics, 2008 ISBN 978-3-540-70776-9 Vol. 152. Jaroslaw Stepaniuk Rough – Granular Computing in Knowledge Discovery and Data Mining, 2008 ISBN 978-3-540-70800-1 Vol. 153. Carlos Cotta and Jano van Hemert (Eds.) Recent Advances in Evolutionary Computation for Combinatorial Optimization, 2008 ISBN 978-3-540-70806-3 Vol. 154. Oscar Castillo, Patricia Melin, Janusz Kacprzyk and Witold Pedrycz (Eds.) Soft Computing for Hybrid Intelligent Systems, 2008 ISBN 978-3-540-70811-7 Vol. 155. Hamid R. Tizhoosh and M. Ventresca (Eds.) Oppositional Concepts in Computational Intelligence, 2008 ISBN 978-3-540-70826-1 Vol. 156. Dawn E. Holmes and Lakhmi C. Jain (Eds.) Innovations in Bayesian Networks, 2008 ISBN 978-3-540-85065-6

Vol. 159. Dimitri Plemenos and Georgios Miaoulis (Eds.) Artificial Intelligence Techniques for Computer Graphics, 2009 ISBN 978-3-540-85127-1 Vol. 160. P. Rajasekaran and Vasantha Kalyani David Pattern Recognition using Neural and Functional Networks, 2009 ISBN 978-3-540-85129-5 Vol. 161. Francisco Baptista Pereira and Jorge Tavares (Eds.) Bio-inspired Algorithms for the Vehicle Routing Problem, 2009 ISBN 978-3-540-85151-6 Vol. 162. Costin Badica, Giuseppe Mangioni, Vincenza Carchiolo and Dumitru Dan Burdescu (Eds.) Intelligent Distributed Computing, Systems and Applications, 2008 ISBN 978-3-540-85256-8 Vol. 163. Pawel Delimata, Mikhail Ju. Moshkov, Andrzej Skowron and Zbigniew Suraj Inhibitory Rules in Data Analysis, 2009 ISBN 978-3-540-85637-5 Vol. 164. Nadia Nedjah, Luiza de Macedo Mourelle, Janusz Kacprzyk, Felipe M.G. Fran¸ca and Alberto Ferreira de Souza (Eds.) Intelligent Text Categorization and Clustering, 2009 ISBN 978-3-540-85643-6 Vol. 165. Djamel A. Zighed, Shusaku Tsumoto, Zbigniew W. Ras and Hakim Hacid (Eds.) Mining Complex Data, 2009 ISBN 978-3-540-88066-0 Vol. 166. Constantinos Koutsojannis and Spiros Sirmakessis (Eds.) Tools and Applications with Artificial Intelligence, 2009 ISBN 978-3-540-88068-4 Vol. 167. Ngoc Thanh Nguyen and Lakhmi C. Jain (Eds.) Intelligent Agents in the Evolution of Web and Applications, 2009 ISBN 978-3-540-88070-7

Vol. 157. Ying-ping Chen and Meng-Hiot Lim (Eds.) Linkage in Evolutionary Computation, 2008 ISBN 978-3-540-85067-0

Vol. 168. Andreas Tolk and Lakhmi C. Jain (Eds.) Complex Systems in Knowledge-based Environments: Theory, Models and Applications, 2009 ISBN 978-3-540-88074-5

Vol. 158. Marina Gavrilova (Ed.) Generalized Voronoi Diagram: A Geometry-Based Approach to Computational Intelligence, 2009 ISBN 978-3-540-85125-7

Vol. 169. Nadia Nedjah, Luiza de Macedo Mourelle and Janusz Kacprzyk (Eds.) Innovative Applications in Data Mining, 2009 ISBN 978-3-540-88044-8

Nadia Nedjah Luiza de Macedo Mourelle Janusz Kacprzyk (Eds.)

Innovative Applications in Data Mining

123

Nadia Nedjah

Prof. Janusz Kacprzyk

Universidade do Estado do Rio de Janeiro Faculdade de Engenharia sala 5022-D Rua S˜ao Francisco Xavier 524 ˜ 20550-900, MARACANA-RJ Brazil Email: [email protected]

Systems Research Institute Polish Academy of Sciences Ul. Newelska 6 01-447 Warsaw Poland Email: [email protected]

Luiza de Macedo Mourelle Universidade do Estado do Rio de Janeiro Faculdade de Engenharia sala 5022-D Rua S˜ao Francisco Xavier 524 ˜ 20550-900, MARACANA-RJ Brazil Email: [email protected]

ISBN 978-3-540-88044-8

e-ISBN 978-3-540-88045-5

DOI 10.1007/978-3-540-88045-5 Studies in Computational Intelligence

ISSN 1860949X

Library of Congress Control Number: 2008935501 c 2009 Springer-Verlag Berlin Heidelberg This work is subject to copyright. All rights are reserved, whether the whole or part of the material is concerned, specifically the rights of translation, reprinting, reuse of illustrations, recitation, broadcasting, reproduction on microfilm or in any other way, and storage in data banks. Duplication of this publication or parts thereof is permitted only under the provisions of the German Copyright Law of September 9, 1965, in its current version, and permission for use must always be obtained from Springer. Violations are liable to prosecution under the German Copyright Law. The use of general descriptive names, registered names, trademarks, etc. in this publication does not imply, even in the absence of a specific statement, that such names are exempt from the relevant protective laws and regulations and therefore free for general use. Typeset & Cover Design: Scientific Publishing Services Pvt. Ltd., Chennai, India. Printed in acid-free paper 987654321 springer.com

Preface

Data mining consists of attempting to discover novel and useful knowledge from data, trying to ﬁnd patterns among datasets that can help in intelligent decision making. However, reports of real-world case studies are not generally detailed in the literature, due to the fact that they are usually based on proprietary datasets, making it impossible to publish the results. This kind of situation makes hard to evaluate, in a precise way, the degree of eﬀectiveness of data mining techniques in real-world applications. On the other hand, rea=searchers of this ﬁeld of expertise usually exploit public-domain datasets. This volume oﬀers a wide spectrum of research work developed for data mining for real-world application. In the following, we give a brief introduction of the chapters that are included in this book. In Chapter 1, the authors present a prototype developed in the context of an Online Satellite Images Distribution System. This prototype have used real Image Data of the Satellite Database and Data Mining techniques to determine the use potentiality of each image received through the Satellite. The obtained results indicate that the use of Data Mining techniques can help in automating the choice of images to be previously processed and stored, representing an improvement in the distribution services rendered and leading to a better utilization of the storage space and the processing resources. In Chapter 2, the authors propose an implementation of the UnBBayes-MEBN framework, which uses the PR-OWL probabilistic ontology language based on Multi Entity Bayesian Networks, as there has been a surge of interest on principled representation of uncertainty applied to the Semantic Web (e.g. W3C’s URW3-XG). The work includes the development of a GUI for building probabilistic ontologies, an API for saving and loading these ontologies, a grammar to specify formulas for creating CPTs dynamically, and a new algorithm for SSBN construction. In Chapter 3, the authors explore the concepts of Multi-objective Metaheuristics (MOMH) and a complete novel approach to induce classiﬁers is described

VI

Preface

where the properties of the rules can be expressed in diﬀerent objectives. Then the algorithm ﬁnds these rules in a unique run by exploring Pareto dominance concepts. Two algorithms are analyzed, a Multi-objective Subset Selection algorithm and a Greedy Randomized Adaptive Search Procedure (GRASP) with path-relinking rule induction algorithm. In Chapter 4, the authors investigate the proposition of a divide-and-conquer approach which is able to detect changes in the underlying distribution of Web usage data. They use a compact structure to summarize the clustering of data within time sub-periods. These summaries are obtained by an evolutionary clustering strategy in order to carry out a follow-up of the user proﬁle over time. In Chapter 5, the authors motivate the KDD practitioners to look beyond data mining in order to infer the characteristics of the data generating process underlying an evolving data base. Changes detected in the patterns mined periodically from an evolving database help in characterizing the underlying data generating process. Detection of changes is preceded by formulating a change model. The chapter describes the change model for streaming itemsets and illustrates the same with the help of two application case studies. In Chapter 6, the authors describe data mining techniques applied to the electric power industry. More speciﬁcally, they compare diﬀerent classiﬁcation algorithms of time series representing short-circuit faults in transmission lines. Results with diﬀerent preprocessing and learning algorithms are presented. It is shown that frame-based sequence classiﬁers can achieve low misclassiﬁcation rates with a reasonable computational cost. We are very much grateful to the authors of this volume and to the reviewers for their tremendous service by critically reviewing the chapters. The editors would also like to thank Dr. Thomas Ditzinger from Springer-Verlag, Germany for their editorial assistance and excellent collaboration to produce this scientiﬁc work. We hope that the reader will share our excitement on this volume and will ﬁnd it useful.

June 2008 Nadia Nedjah Luiza M. Mourelle Janusz Kacprzyk

Contents

1 Application of Data Mining Techniques to Storage Management and Online Distribution of Satellite Images Denise N. Rotondi Azevedo, Jos´e M. Parente de Oliveira . . . . . . . . . . . . . . .

1

2 A GUI Tool for Plausible Reasoning in the Semantic Web Using MEBN Rommel N. Carvalho, Marcelo Ladeira, La´ecio L. Santos, Shou Matsumoto, Paulo Cesar G. Costa . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

17

3 Multiobjective Optimization and Rule Learning: Subselection Algorithm or Meta-heuristic Algorithm? Celso Y. Ishida, Aurora Pozo, Elizabeth Goldbarg, Marco Goldbarg . . . . . .

47

4 Clustering Dynamic Web Usage Data Alzennyr da Silva, Yves Lechevallier, Fabrice Rossi, Francisco de Carvalho . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

71

5 Towards Characterization of the Data Generation Process Vasudha Bhatnagar, Sarabjeet Kochhar . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

83

6 Data Mining Applied to the Electric Power Industry: Classiﬁcation of Short-Circuit Faults in Transmission Lines Yomara Pires, Jeﬀerson Morais, Claudomir Cardoso, Aldebaro Klautau . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107 Author Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123

List of Figures

1.1 1.2 1.3 1.4 1.5 1.6 1.7 2.1 2.2 2.3 2.4 2.5 2.6 2.7 2.8 2.9 2.10 2.11 2.12 2.13 2.14 2.15 2.16 2.17 2.18

Data, information and Knowledge pyramid . . . . . . . . . . . . . . . . . . . CBERS - Browse Image . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Delivery Request Time . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Satellite Data Processing System . . . . . . . . . . . . . . . . . . . . . . . . . . . . Catalogue Database . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Numeric Attributes - Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Nominal Attributes - Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . An example of a Starship MFrag . . . . . . . . . . . . . . . . . . . . . . . . . . . . A code example to produce a CPT . . . . . . . . . . . . . . . . . . . . . . . . . . The Starship MTheory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . The Star Trek Decision MFrag . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . SSBN for the Star Trek MTheory with Four Starships within Range . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Concepts of the PR-OWL language . . . . . . . . . . . . . . . . . . . . . . . . . . PR-OWL ontology elements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Node speciﬁcation with Protege . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Node speciﬁcation with UnBBayes-MEBN . . . . . . . . . . . . . . . . . . . . General view of the MTheory and MFrag in UnBBayes . . . . . . . . . Main MEBN elements design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Main MEBN Entity and Type design . . . . . . . . . . . . . . . . . . . . . . . . Knowledge Base and PowerLoom design . . . . . . . . . . . . . . . . . . . . . . SSBN construction for HarmPotential(!ST4,!T3 ) query - Step 1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . SSBN construction for HarmPotential(!ST4,!T3 ) query - Step 2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . SSBN construction for HarmPotential(!ST4,!T3 ) query - Step 4 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . SSBN construction for HarmPotential(!ST4,!T3 ) query - Step 5 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . SSBN construction for HarmPotential(!ST4,!T3 ) query - Step 9 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

2 5 5 6 7 12 13 20 21 26 27 29 31 32 33 34 34 36 37 38 39 40 40 41 41

X

List of Figures

2.19 SSBN construction for HarmPotential(!ST4,!T3 ) query - Step 18 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.20 SSBN generated for the HarmPotential(!ST4,!T3 ) query . . . . . . . 2.21 Grammar used for dynamically generating a CPT . . . . . . . . . . . . . 3.1 Example of approximation set of positive rules for data set #2 (bupa) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.2 Approximation sets of positive rules for data set #2 (bupa) . . . . . 4.1 Global clustering . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.2 Independent local clustering . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.3 Previous local clustering . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.4 Dependent local clustering . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.5 Contingency table . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.6 Corrected Rand index values computed partition by partition . . . 4.7 F-measure values computed cluster by cluster . . . . . . . . . . . . . . . . . 5.1 Tasks in the process of characterizing data generation process . . . 5.2 Architecture of DGP-CHAR System . . . . . . . . . . . . . . . . . . . . . . . . . 5.3 Consolidation at level 0 and 1, with windows of sizes s1 = 3 and s2 = 2 respectively . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.4 The SI-CHAR System . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.5 Support behavior of an itemset in partitioned support space for time interval [t1 , t5 ] . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.6 Computation of PF of an item-set x . . . . . . . . . . . . . . . . . . . . . . . . . 5.7 Support of Proquest . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.8 Support of Springerlink . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.9 Support of itemset {Dead, Male} . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.10 Support of itemset {Dead, Female} . . . . . . . . . . . . . . . . . . . . . . . . . 5.11 Support of itemset {Morning rush hour, Seriously injured} . . . . . 5.12 Support of itemset {Evening rush hour, Seriously injured} . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.1 The input and output matrices of the front end stage. Q and K are the dimension of the sample and feature vectors, respectively. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.2 Organizing feature vectors z in a concatenated raw front end. In this case, the two faults with a total of three frames and L = 2 lead to three vectors z of dimension K = 12. . . . . . . . . . . . . 6.3 Block for simulating a fault. The SW elements are switches and R are resistances. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.4 Sample complexity for frame-based classiﬁcation (the error is Ef ) using ANN. The left-most ﬁgure shows the individual results for each of the 5 disjoint datasets and the right-most shows the average and standard deviation. It can be seen that approximately M = 700 examples suﬃces to train this classiﬁer. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

42 42 44 60 61 74 74 75 76 78 79 80 85 87 89 91 92 93 98 98 101 101 102 102

110

112 115

118

List of Figures

6.5

6.6 6.7 6.8

Error rate Ef for several classiﬁers and frame lengths (L) using the concatenated raw front end and allfault normalization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Error Es for post-fault classiﬁcation. The ANN-based FBSC achieved Es = 0 for L > 1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Diﬀerence Ef − Es between the error rates for frame-by-frame and sequence classiﬁcation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Results (Ef ) for two wavelet front ends using the ANN and J4.8 classiﬁers. The best result with the concatenated raw front end is also shown, for the sake of comparison. . . . . . . . . . . . .

XI

119 119 120

120

List of Tables

1.1 1.2 1.3 1.4 1.5 3.1 3.2 3.3 3.4 3.5 3.6 3.7

3.8

4.1 5.1 5.2 5.3 5.4 5.5 5.6 6.1

Scenes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Clustering classes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . J48 - Numeric attributes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . J48 - Nominal attributes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Incorrect classiﬁcation index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . A contingency table . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Description of data sets used in the experiments . . . . . . . . . . . . . . . Mean AUC (Other Systems) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Mean AUC PF Elite . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Mean Support and Weighted Relative Accuracy . . . . . . . . . . . . . . . Mean AUC for GRASP-PR Rule Learning with 100 and 200 of maximal contructed solutions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Number of times that PFE with no Minimum parameters has better aproximation sets of positive and negative rules than GRASP-PR Rule Learning algorithm with parameter maxLoop = 100 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Number of times that PFE with no Minimum parameters has better Pareto Front of positive and negative rules than GRASP-PR Rule Learning algorithm with parameter maxLoop = 200 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Description of the variables describing navigations . . . . . . . . . . . . . Change metrics for Proquest database . . . . . . . . . . . . . . . . . . . . . . . Change metrics for Springerlink database . . . . . . . . . . . . . . . . . . . . Change metrics for itemset {Dead, Male} . . . . . . . . . . . . . . . . . . . . Change metrics for itemset {Dead, Female} . . . . . . . . . . . . . . . . . . Change metrics captured by the online component for itemset {Morning rush hour, Seriously injured} . . . . . . . . . . . . . . . . . . . . . . Change metrics captured by the online component for itemset {Evening rush hour, Seriously injured} . . . . . . . . . . . . . . . . . . . . . . Summary of parameters for the front ends and two classiﬁers . . .

8 10 11 12 13 48 53 54 55 56 64

65

66 77 99 99 101 101 102 102 118

1 Application of Data Mining Techniques to Storage Management and Online Distribution of Satellite Images Denise N. Rotondi Azevedo1 and Jos´e M. Parente de Oliveira2 1

2

National Institute for Space Research [email protected] Technological Institute of Aeronautics [email protected]

The evolution of computational systems and data capacity storage brought a spread of information to many society niches. This information, normally heterogeneous and dispersed in the organizations, can bring in itself a set of indirect knowledge with a great potential of use. The knowledge extracted from this information can be valuable for proﬁle evaluations, detection of problems and opportunities, in the decision making, etc. In this context, the concepts of Knowledge Discovery in Databases (KDD) came up, having, as part of the process, the Data Mining techniques which have as their main objective patterns discovering in large data sets. The Data Mining algorithms and techniques deal with two dimensions of problems: the great amount of data that makes any discovering process of known patterns very complex and, on the other hand, the search of a priori unknown patterns that can be very useful in organizations. This chapter describes the use of knowledge discovery techniques by means of the use of traditional Data Mining algorithms in a real context of a Satellite Images Processing and Distribution System by Internet. The system described stores about 450,000 images and distributes an average of 10,000 images a month, usually delivering images to the users in up to 4 minutes. However, in some cases, this delivery time can take up to 20 minutes or more. This discrepancy in the delivery time occurs whenever a user requests an image that has not been processed or has not been requested previously. In this case, the user has to wait for the image processing. Today, the choice criteria of the images that will be processed a priori are not automated and depend on the operators who take this decision based on empirical criteria. The use of Data Mining techniques has helped in automating the choice of which images would be previously processed and stored. It represented an improvement in the customer service and leaded to a better use of the storage space and the processing resources. The chapter is organized as follows. Section 1.1 presents the concepts used in the chapter. Section 1.2 presents a characterization of the research problem N. Nedjah et al. (Eds.): Innovative Applications in Data Mining, SCI 169, pp. 1–15. c Springer-Verlag Berlin Heidelberg 2009 springerlink.com

2

D.N.R. Azevedo and J.M.P. de Oliveira

considered. Section 1.3 describes how data mining techniques have been applied. Section 1.4 presents some concluding remarks.

1.1 Introduction Data Mining can be seen as a stage of a larger process known as Knowledge Discovery in Databases (KDD). A known deﬁnition for the knowledge discovery is [2]: ”KDD is a process, in several stages, not trivial, interactive and iterative, for identiﬁcation of new valid understandable and potentially useful patterns from large data sets”.

Fig. 1.1. Data, information and Knowledge pyramid

The data, information and knowledge concepts can be seen in the following Figure 1.1 [3]. Thus, the use of Data Mining is intended to support the discovery of patterns in databases in order to transform information in knowledge [3], to assist the decision making process or to explain and justify it. According to Witten and Frank [4], Data Mining can be deﬁned as an automatic or semiautomatic patterns discovery in great amounts of data, where these patterns can be perceived as useful. Some authors divide the complete process of data mining in the following stages: problem deﬁnition, application of Data Mining techniques (prototyping), implementation and evaluation of investment return [3][1]. The stage of data mining application can still be divided in three great iterative stages: pre-processing, pattern extraction and post-processing [3][5]. 1.1.1

Pre-processing

The data mining is usually carried out on great and complex masses of data that are centralized in one database, dispersed through several databases, or

1 Application of Data Mining Techniques to Storage Management

3

distributed in many formats and/or medias. In any case, a pre-processing stage is necessary. This stage should be based on a deep analysis of the problem, context and available data. Many tasks must be carried out and some decisions must be taken. The main tasks in pre-processing are: a) Extraction and integration: dispersed data in diﬀerent sources and medias must be integrated and extracted, tables in databases may be joined and ”desnormalized”; b) Transformations: data can be codiﬁed, normalized, enriched with external information, etc. aiming at a better adjustment to the problem goal, data mining algorithms and techniques or to the processing requirements. c) Cleaning: the data can contain incorrectness and it should be completed or cleaned to prevent absent data or to inconsistent data (outliers)[3]. d) Data Selection and reduction: most of the time it is not needed to use all the available data items. Sometimes the use of the complete amount of data, or the use of all existing attributes can aﬀect the processing and even the results in a bad way. So, selections and reductions can be applied in the attributes, in the attributes space and in the sample number. The problem analysis and the pre-processing stages are very important and sensitive in the Data Mining Process. A wrong decision in data choice, or the use of incomplete attributes, or any deformation in the data can invalidate the patterns extraction or can even take to inconsistent patterns. 1.1.2

Pattern Extraction

This stage is the mining process itself. Normally, the characteristic of the problem, its domain and expected results deﬁne the model to be used for the pattern extraction. The models can be predictive or descriptive. ”A predictive Model calculates some value that represents a level in the future activity, a descriptive model discovers rules that are used to group items in categories”[1]. Therefore, the problem classiﬁcation, the expected results and the characteristics of the data must guide the choice of the mining techniques and the most appropriate algorithms. The main mining techniques are: a) Classiﬁcation: prediction of a category or discrete class. From classes already known the classiﬁcation algorithm can be trained to characterize new records for predicting its category. Several algorithms are used: C.45 (decision tree), Network Bayes, Neural Networks, etc; b) Regression: it is also associated to prediction, and has similar objectives to the classiﬁcation, but it is applied to continuous values. This technique aims to seek a function that maps values of a set of attributes in a target attribute value, so that it can estimate the value of this target attribute in future samples. The regressions can be linear or nonlinear. Statistical algorithms and neural networks are used in this task;

4

D.N.R. Azevedo and J.M.P. de Oliveira

c) Association: related to descriptive models, it looks for the identiﬁcation of patterns that occur simultaneously in collections of data; d) Clustering: technique of samples aggregation that has as main objective to maximize the intra-groups similarity and to minimize the inter-groups similarity [3]. 1.1.3

Post-processing

Once realized the pattern extraction, in the last stage, the Post-processing, the results are analyzed. Many times, diﬀerent patterns are found in data and a deep analysis of the results is needed in order to ﬁlter the less important or very abnormal results. Usually, a problem domain expert conducts this analysis. The data mining is intended to validate hypotheses or to look for new patterns inside the data. Thus, for this purpose, it must be veriﬁed: • If patterns had been found according to the domain specialist’s expectations; • Which are the diﬀerences between the specialist’s knowledge and the found patterns; • If the new patterns found are relevant, logic and useful. It is important to remind that data mining is in fact an experimental process and, therefore, the stages should not be completely tight, they should be iterative.

1.2 Problem Deﬁnition The Data Center of the National Institute for Space Research in Brazil has a satellite image data base (CBERS and Landsat Satellites) of approximately 120 Tbytes with a growing rate of about 30 Gbytes a day. These data - raw data and products - are stored in a Library and are addressed by an image Catalogue which maintains for each satellite image a metadata that describes it and a sub-sampled image (browse image) for its visualization. Through this catalogue the user can search images, request the desired images and make download of the chosen ones - a product request. Figure 1.2 shows CBERS sub-samples images that are visualized by users in the catalogue. Using a Catalogue System, users may put requests for a product (processed scene). When the product is ready for download, an e-mail is sent to the user advising that he/she can download the processed scene. Currently, a user receives an ordered product in a short period of time, elapsing, most of the time, up to 4 minutes between the moment of the request and the reception (download) of the image (0-4 minutes: 46% of the cases). Figure 1.3 summarize the delivery times. This eﬃciency is due to some factors: use of distributed processing in different stages of the process; use of an automatic Library to store the data, pre-generation of products following certain established criteria or by demand;

1 Application of Data Mining Techniques to Storage Management

5

Fig. 1.2. CBERS - Browse Image

Fig. 1.3. Delivery Request Time

and the storage of part of processed images in disk (online access) - the other part of processed images are usually stored in Library tapes (near line access). The entire process can be described in the following steps, as can be seen in Figure 1.4. 1. At the Reception Station: a) The raw satellite data is received using an antenna; b) The raw data are sent to the Processing Station. 2. At the processing station: a) Using the Catalogue Generation Sub-system, the received raw satellite data are stored in a library and processed to generate the catalogue database (metadata and browse images); b) Using the Pre-processing sub-system, the raw data are preprocessed and stored in disks. As there is a limit in the disk space, it is kept just the

6

D.N.R. Azevedo and J.M.P. de Oliveira

Fig. 1.4. Satellite Data Processing System

images for the last month of reception. The reprocessing step may cut the product generation time in a half; c) And, as the last step, using deﬁned criteria - normally better images -, part of these images is entire processed and the generated products stored. It makes, for those images, the delivery time immediate. The products are also stored by demand, in other words, only the ﬁrst user that requests a speciﬁc product (image) has to wait for the processing time. Using this process, although in most of the time the delivery of a request is almost immediate, for some requests this delivery can take up to 4 hours. Additionally, a lot of images that would never be requested or that would be requested just one time are being pre-processed, entirely processed or stored in the disk causing processing and storage waste. The main objective of the work developed has been to maximize the speed in the attendance of a user request, through the pre-processing of images with greater potential of use, aiming at the optimization of the employed resources - processing and storage. To accomplish such an objective, it has been required the study of usage behaviors of satellite images through Data Mining techniques and the correspondent classiﬁcation of these images in relation to the identiﬁed usage behaviors.

1.3 Application of Data Mining Techniques After the deﬁnition of the problem, among the several approaches found in the literature, a prototype was deﬁned where the following iterative phases for the work have been identiﬁed:

1 Application of Data Mining Techniques to Storage Management

• • • • •

7

Deﬁnition of the environment and tools to be used; Deﬁnition of the data mining techniques and algorithms to be used; Data pre-processing; Patterns extraction using the deﬁned tool; Post-processing and analysis of the results.

Afterwards, it is described in details the results of each data mining application stage. 1.3.1

Deﬁnition of the Used Environment and Tools

For the development of the prototype, the computational learning environment Weka (Waikato Environment Knowledge Analysis), which implements a relatively complete set of Data Mining algorithms, has been used. This tool has interfaces where diﬀerent algorithms can be exercised interactively and a graphical support that aids in the visualization of results and data. It has also an interface to Java language that has been used in the phase of prototype implementation. Mondrian tool has been also used as a support for the graphical analysis of data. 1.3.2

Description of Used Data

The prototype was made using real data from the Image Satellite Database (Catalogue) of National Institute for Space Research. This work used data from 2004 and 2005, as explained in the sampling step. As represented by Figure 1.5, the catalogue data is currently composed by some distinct databases: Users database, Scene database (metadata of the stored scenes) and Request database (requests and their items). Also, mainly the descriptive data of the images (metadata) and the information on its use (number of requests) have been used. From the original databases the following tables have been used: Scene (Scenes DB - 38 attributes), Grid

Fig. 1.5. Catalogue Database

8

D.N.R. Azevedo and J.M.P. de Oliveira

(Scenes DB - 6 attributes), Request (Request DB - 14 attributes) and RequestItem (Request DB - 22 attributes). 1.3.3

Description of Applied Pre-processing

In the development, some databases tables, using a speciﬁc program, have been pre-processed and converted generating Arrf ﬁles (format used by the Weka software). In this ﬁrst conversion stage, several tables of the diﬀerent databases have been joined and related in a process of ”denormalization” and some attributes have been discarded. After the analysis of the presented data, it was decided for another set of modiﬁcations, reductions and enrichment of the data aiming at better adjusting them to the algorithms that would be used and to the deﬁned software structures. The following processing tasks have been carried out: • The attribute Season was added because of the interest in the seasonality and to facilitate the application of the classiﬁcation algorithms that could supply information on the data characteristics. The experience showed that some phenomena are seasonal as, for example, agriculture of some craps that happens in speciﬁc months of the year, the cloud months in the forest, and so on; • To avoid that the cloud coverage indexes for a quadrant (quarter of scene) could have an incorrect eﬀect in the classiﬁcation results, increasing the decision tree complexity, it has been adopted the attribution of just one cloud cover index for the whole scene (an average). This index was reduced to discrete values from 0 to 10; • In relation to the Cloud Coverage, it was kept also the lesser quadrant index. This information was kept trying to minimize the eﬀect caused by the use of Table 1.1. Scenes Attribute Sensor Path Row Region Month Season Indicative of Earth or Water

Description Satellite Instrument Position in a grid Position in a grid Geographic region Acquisition month (image date) Acquisition season An indicative if the scene was over sea or earth Cloud cover index of the scene (0Cloud Coverage Index (average) 10) Cloud cover index of the lesser Cloud Coverage Quadrant cloudy quarter of scene Classes of use Scenes Class (Used, Not Used) Scenes Class (Not used, Used, Very Classes of use Used)

Type Nominal Numeric Numeric Nominal Numeric Nominal Nominal Numeric Numeric Nominal Nominal

1 Application of Data Mining Techniques to Storage Management

9

the average. An user may request a cloudy scene because the area of interest is in a part of the image that is clear, in this case it is possible that the average index is bad but this scene is still interesting for the user; • For the ﬁnal classiﬁcation of a scene, two approaches were analyzed: the use of classes Not Used and Used to indicate the use or not of an image and the use of classes Used, Not Used and Very Used, quantifying the use and trying to determine images with greater potential of interest. Table 1.1 shows the ﬁnal version for the Image Table attributes (Scenes). 1.3.4

Sampling

Analyzing the data base used, at the moment of processing, there were 100,352 valid images where 77.69% were Not Used images, 16.87% were images Used (up to 10 requests) and 5.44% were images Very Used (more than 10 requests). From these data two types of sampling has been carried out: • Random data sampling by class: 50% of images Not Used (38,983 scenes), 60% of images Used (10,156 scenes) and 80% of images Very Used (4,365 scenes), a total of 53,504 scenes. From these scenes two ﬁles were generated, one for training the algorithms and another for validation, in the ratio of 2 registers for training and 1 register for validation as Rezende advises [5]; • Sampling per year: all images of the year 2004 (44,647 images) have been used for the training set and a random sampling of scenes of the year 2005 (18,478 scenes) has been used for the validation set. 1.3.5

Application of Data Mining Algorithms and Results

The application of data mining algorithms and their results is a macro step that can be divided into sub-steps: • Deﬁnition of Data Mining techniques and algorithms: in this sub-step, after the data analysis, the problem is identiﬁed and the algorithms to be applied are selected; • Using the deﬁned patterns extraction tool: in this sub-step the algorithms chosen to be tested are eﬀectively applied to the data through the chosen tool and patterns are obtained, compared and ﬁltered; • Results: in this latter sub-step, the obtained patterns are consolidated and analyzed. Deﬁnition of Data Mining Techniques and Algorithms For solving the identiﬁed and delimited problem, it was necessary to discover the potential use of the images which would help to get to an automatic criteria for identiﬁcation of the scenes that should be pre-processed or not, and that should be stored at the disk or not (online), aiming at the customer service improvement without processing overload and with storage optimization. This

10

D.N.R. Azevedo and J.M.P. de Oliveira

problem was identiﬁed as being a classiﬁcation problem. Thus, the scenes have been classiﬁed in function of its use and two approaches already described were adopted: Scenes were divided into two groups, where the ﬁrst one contained Used and Not Used scenes and second one contained Not Used, Used and Very Used scenes. The objective was, at the ﬁrst moment, to understand which attributes were important in the categorization of the use of a scene and, after that, to verify if it was possible to classify scenes in agreement with these attributes for the chosen category. An algorithm that implements a decision tree has been chosen, since this kind of tree was capable of explaining which attributes were important, how it classiﬁed the data and how it inﬂuenced in the determination of the usability of a scene. According to Witten [4], the data explanation is so important or even more important than its classiﬁcation. Weka software and the J4.8 algorithm that is an implementation of C4.5 algorithm have been used. The C4.5 algorithm in turn is the ID3 - a classiﬁer that generates a decision tree to classify the test instances - with some modiﬁcations. After that, it was used also a neural network algorithm to classify the data and a comparison of the results was made. Patterns Extraction Using the Deﬁned Tool As the ﬁrst step, before the application of the Classiﬁcation algorithms, aiming at validating the chosen classes, the clustering algorithm SimpleKMeans’s implemented by the Weka tool was applied. The obtained results are shown in Table 1.2. Table 1.2. Clustering classes Class Attribute Samples Incorrectly Error Classiﬁed (%) (Clusters) (0) NotUsed (1) Used 35,673 14,809 41.51 20,428 57.26 (0) NotUsed (1) Used 35,673 (2) VeryUsed

Then, the J48 algorithm was executed with pruning (a cut in the number of instances in the leaves - minimum of 10 and 50 instances). This choice didn’t bring great loss in the classiﬁcation capability or in the obtained information since the number of instances was relatively large (44,647 training instances). For the decision tree training, two experiments with attributes that were equivalent in semantic but diﬀerent in type and granularity have been carried out. The ﬁrst one prioritized the original numeric attributes: the location in a geographical grid (numerical path and row) and the month of the year. The second experiment used nominal attributes, where the spatial references were the geographical areas that included a set of paths and rows, and the temporal

1 Application of Data Mining Techniques to Storage Management

11

Table 1.3. J48 - Numeric attributes

references were the seasons representing about three months each. It was expected with this second experiment made with the transformed attributes, the obtainment of a data use characteristics description. A summary of the obtained results is showed in Table 1.3 and Table 1.4. Observing the tables, it can be noticed that the ﬁrst experiment showed a tree with better index of classiﬁcation in the training stage, however larger and more complex; while the second experiment resulted in a tree with a little increase in the classiﬁcation error, however smaller and more explanatory. It could still be observed that, although the percentage of scenes classiﬁed correctly was larger in the case of the numeric attributes, the second experiment classiﬁed the test sample better. It is worth pointing out that the second experiment used more generic attributes and a larger pruning in the tree. This might have made possible a better categorization of the new scenes. The tables also show, through the classiﬁcation index per classes, that the correctness classiﬁcation index of Not Used scenes was above the average of the general classiﬁcation correctness, and that there was a great dispersal in the results of class Used (less than 10 downloads). They still show that the classiﬁcation error in the scenes Very Used occurs mainly in the Used scenes column. As the problem focused mainly on the scenes Not Used and Very Used

12

D.N.R. Azevedo and J.M.P. de Oliveira

Table 1.4. J48 - Nominal attributes

Fig. 1.6. Numeric Attributes - Results

these data characteristics presented by the decision tree did not disturbed the expected results. Figure 1.6 and 1.7 summarize and consolidate the results. For the choice of numeric attributes (geographical grid and the month of the year), Figure 1.6 shows graphically the percentage of samples that have been corrected (blue) an incorrect (red) classiﬁed in both phases: training phase and testing phase. Figure 1.7 shows the same information for nominal attributes (geographical areas and seasons).

1 Application of Data Mining Techniques to Storage Management

13

Fig. 1.7. Nominal Attributes - Results

A Weka implementation of Neural Network algorithms were also applied, using the following conﬁgurations for the Network: 5 neurons in one hidden layer, 25 neurons in one hidden layer and 25 and 5 neurons in two hidden layers. The best results obtained with these algorithms are described next. Results Some considerations to the application of the algorithms are the following: Neural Network Algorithm. The results of the application of Neural Network algorithms, for many diﬀerent designs of the network, were very similar to the J48 results. As the decision tree explains better the data and had better performance, the decision tree has been chosen for the prototype. Decision Trees Algorithm. Although the application of the clustering algorithm has presented a better result when it considered two classes (Used and Not Used), and, in consonance, the decision tree has classiﬁed the images with larger precision, nevertheless, it was preferred to use three classes (Used, Not Used and Very Used), as explained above. Table 1.5 shows the incorrect classiﬁcation index: Table 1.5. Incorrect classiﬁcation index Class

Incorrect Classiﬁed As Very Used As Used As Not Used Very Used ( 7%) 22.74% 1.91% Used ( 18%) 5.07% 32.08% Not Used ( 75%) 0.17% 3.71%

Because of the data characteristics, it was obtained a better result of the decision tree for the training set when all scenes of whole year (2004) were used. The generated tree was tested with a random scenes set of the year 2005.

14

D.N.R. Azevedo and J.M.P. de Oliveira

From the decision tree generated through the nominal attributes, some inferences concerning the data use characteristics could be made: • The main cut in the tree was made on cloud coverage index in two main points: less than 30% with larger percentage of Used and Very Used scenes and above 80% with most of the scenes Not Used. These values - 30% and 80% of coverage - brought quite important information enriching the intuitive choice criteria; • Another important cut in the tree was the indicative of Water or Land or if scenes are inside the country or not (Region = Other); • 10% or less for the cloud coverage index normally classiﬁes the scenes as Very Used; • When season was winter, for the same values in the other attributes, the images were always classiﬁed as higher level than the other season’s images. Many inferences made using the results were in agreement and close to the known patterns in the data behavior and, sometimes it is interesting to prove patterns that are known by the experience, or logic, or even intuitively through an algorithm information, but in this case, the values and limits found are more precise and can be useful in decision making over the data. It was obtained an index of about 97% of correctness for the scenes that certainly will never be requested and an index of about 73% for the scenes that probably will be used many times and has to be in an online archive. Comparing the results with the criteria previously being used, the work realized can: • Provide an economy of: 1. About 50% in processing time; 2. About 10% in the storage space (12 Tbytes). • Improve the delivery time: 1. The number of requests delivered in 0 minutes has been increased to 41% of the total (today is 17%); 2. The number of requests delivered in 9 minutes has been increased to 43% of the total (today is 36%); 3. The number of requests delivered in more than 9 minutes has been decreased to 16% of the total (today is 47%).

1.4 Conclusion This chapter presented the use of Data Mining algorithms in a real context of a Satellite Images Processing and Distribution System by Internet. A prototype using Data Mining techniques, mainly classiﬁcation algorithms, was developed in order to determine the use potentiality of each image received through the Satellite, automating the choice criteria of each image to be preprocessed or entirely processed a priori.

1 Application of Data Mining Techniques to Storage Management

15

The main objective with the use of data mining techniques is the improvement of the distribution services rendered, as well as the improvement of the processing resources and storage administration. The results obtained from the pre-processed data presented to the algorithms showed patterns that are close to the patterns already known, and the analysis of the decision tree emphasized some characteristics in the classiﬁed data, bringing some new and important knowledge of values for some known patterns. As already said, for the considered problem, the use of a classiﬁer algorithm, especially if centered in the Not Used and Very Used images, seemed to be quite interesting in the election of scenes to be discarded or processed/pre-processed and in the management of the data storage (online and near line images). The application of Data Mining techniques allows an intelligent automation of the process, bringing economy in the processing eﬀort, in the storage space, and decreasing the request delivery time of images.

References 1. Braga, L.P.V.: Introdu¸ca ˜o a ´ Minera¸ca ˜o de Dados, 2. Edi¸ca ˜o revista e ampliada, Rio de Janeiro: E-Papers Servi¸cos Editoriais, p. 212 (2005) 2. Fayyad, B.M., Piatetsky-Shapiro, G., Smyth, P.: From Data Mining to Knowledge Discovery: An Overview. In: Knowledge Discovery and Data Mining. AAAI Press, Menlo Park (1996) 3. Goldschmidt, R., Passos, E.: Data Mining - Um Guia Pr´ atico. Elsevier, Rio de Janeiro (2005) 4. Witten, H.I., Frank, E.: Data Mining - Practical Machine Learning Tools Techniques and storage diseases, 2nd edn. Elsevier, MA (2005) 5. Rezende, S.O.: Sistemas Inteligentes: Fundamentos e Aplica¸co ˜es. Manole, S˜ ao Paulo (2003)

2 A GUI Tool for Plausible Reasoning in the Semantic Web Using MEBN Rommel N. Carvalho1 , Marcelo Ladeira1 , La´ecio L. Santos1 , Shou Matsumoto1 , and Paulo Cesar G. Costa2 1

2

Computer Science Department - University of Brasilia - Brasilia, DF - Brazil {rommel.carvalho,laecio,cardialfly}@gmail.com, [email protected] Center of Excellence in C4I - George Mason University - Fairfax, VA - USA [email protected]

As the work with semantics and services grows more ambitious in the Semantic Web community, there is an increasing appreciation on the need for principled approaches for representing and reasoning under uncertainty. Reacting to this trend, the World Wide Web Consortium (W3C) has recently created the Uncertainty Reasoning for the World Wide Web Incubator Group (URW3-XG) to better deﬁne the challenge of reasoning with and representing uncertain information available through the World Wide Web and related WWW technologies. In according to the URW3-XG eﬀort this Chapter presents the implementation of a graphical user interface (GUI) for building probabilistic ontologies, an application programming interface (API) for saving and loading these ontologies and a grammar proposal to specify formulas for creating conditional probabilistic tables dynamically. The language used for building probabilistic ontologies is Probabilistic OWL (PR-OWL), an extension for OWL based on Multi-Entity Bayesian Network (MEBN). The GUI, API, and the compiler for the proposed grammar were implemented into UnBBayes-MEBN, an open source, Java-based application that provides an easy way for building probabilistic ontologies and reasoning based on the PR-OWL/MEBN framework.

2.1 Introduction Writing has been the dominant way of representing and communicating until the second half of the last century, when the digital computing became the main force of what Alvin Toﬄer [14] called the “Third Wave” (the ﬁrst being the agricultural revolution and the second being the industrial revolution). In this Third Wave we can see the “information technology revolution”. However, this phase is coming to its end with the arrival of what we call “knowledge revolution” that emerges as a natural subsequent phase of the Third Wave. The “knowledge revolution” will be seen, in the future, as the phase where the arduous and manual task of identifying, accessing and utilizing information was successfully assigned to computers, allowing human beings to change their N. Nedjah et al. (Eds.): Innovative Applications in Data Mining, SCI 169, pp. 17–45. c Springer-Verlag Berlin Heidelberg 2009 springerlink.com

18

R.N. Carvalho et al.

focus from data driven to knowledge driven activities. Within this context, the Semantic Web (SW) emerges as a collaboration eﬀort between the W3C, various researchers, and industrial partners aimed to both understand the current Web and to provide a common framework for allowing data sharing and reusing among applications, enterprises, and communities. The SW will achieve its full potential when it becomes a place where data can be shared and processed by automatic tools the same way it is currently done by human beings [2]. According to the W3C [5], ontologies are envisioned as the technology providing the cement for building the SW. Ontologies contain a common set of terms for describing and representing a domain in a way that allows automated tools to use stored data in a wiser, context-aware fashion, intelligent software agents to aﬀord better knowledge management, and many other possibilities brought by a standardized, more intensive use of metadata. Unlike what happens in syntacticonly protocols, ontologies allow applications to infer the diﬀerence between the surname White and the color white. Yet, when comparing two ontologies containing the term “White”, current SW deterministic reasoning algorithms will either consider it to be a color or an undeﬁned concept (which is not the same as a non-color), with no intermediate grading. This is acceptable when complete information is available, which is frequently the case under the closed world assumption but much less common in open world environments such as the Web, where available information can be partial (not complete) or approximate (not exact). Thus, it is necessary to have ways of dealing with uncertainty in the SW. In spite of that scenario, current development for SW applications (which includes automated reasoning in most of its activities) is based on classical logic. As an example, OWL, a W3C recommendation [5, 10], has no built-in support for probabilistic representation and reasoning. This is a major shortcoming for a technology that is expected to operate in a complex, uncertain open world environment. The W3C responded to this limitation with the recently created Uncertainty Reasoning for the World Wide Web Incubator group (URW3-XG) [8]. The group’s mission was to better deﬁne the challenge of representing and reasoning with uncertain information within the World Wide Web and its related technologies. The use of probabilistic reasoning enables information systems to derive beneﬁt from uncertain, incomplete information, instead of being restricted to complete knowledge alone. This seems to be a promising prospect for the SW. One of the most promising approaches to deal with uncertainty in the SW is Bayesian networks (BN) [11], a graphical, ﬂexible means of parsimoniously expressing joint probability distributions over many interrelated hypotheses. However, BNs have some limitations on representational power that restricts their use for the SW. Amongst these limitations are the fact that the number of variables has to be known in advance and the technique’s lack of support for recursion. In order to address these shortcomings within the context of the SW, Costa [2] proposed a Bayesian framework to probabilistic ontologies that provides a basis for representation and reasoning under uncertainty with the expressiveness required by SW applications. This framework is based on the probabilistic ontology

2 A GUI Tool for Plausible Reasoning in the Semantic Web Using MEBN

19

language PR-OWL [4, 3], which uses Multi-Entity Bayesian Networks (MEBN) [6, 7] as its underlying logic. MEBN is a formalism that brings together the expressiveness of First-Order Logic (FOL) with BN’s ability to perform plausible reasoning. This Chapter introduces an implementation of MEBN being currently developed at the University of Brasilia (UnB) with support from George Mason University’s C4I Center. Its current feature list includes a GUI for modeling probabilistic ontologies, the ability to save probabilistic ontologies in PR-OWL format, and a MEBN-based reasoning engine. This framework is being implemented as an extension of the UnBBayes, an open source software developed at UnB that is capable of modeling, making inferences and learning probabilistic networks. This Chapter is structured as follows. Section 2.2 introduces the basic concepts of the MEBN formalism. Section 2.3 presents the PR-OWL language. Section 2.4 is dedicated to describe some features of the UnBBayes-MEBN1 and to discuss implementation details, with a focus on creating, loading, and saving PR-OWL ﬁles as well as on logic formulas for representing MEBN conditional probabilistic tables (CPT). These formulas are used to dynamically create CPTs needed to perform probabilistic reasoning.

2.2 An Overview of the MEBN MEBN is a ﬁrst-order Bayesian logic that integrates classical ﬁrst-order logic with probability theory. MEBN represents the world as comprised of entities that have attributes and are related to other entities. Knowledge about the attributes of entities and their relationships with each other is represented as a collection of MEBN fragments (MFrags) organized into MEBN Theories (MTheories). MFrag consists of both a set of CPTs and FOL logical con-straints that establish their validating conditions. The number of random variables (RV) is not ﬁxed in a MEBN model. Instead, RVs are instantiated dynamically. An MTheory is a set of MFrags that satisfy certain FOL consis-tence conditions that guaranty the existence of a unique joint probabilistic distribution (JPD) under its RVs. When all RVs are instantiated, all consistence conditions are satisﬁed, and all CPTs are generated, the MEBN yields a Situation Speciﬁc Bayesian Network (SSBN). An SSBN is a normal BN. As a means to provide a smooth introduction to the fairly complex concepts of MEBN logic, we needed to explore a domain of knowledge that would be easily understood, while still rich enough to include scenarios that would demand a highly expressive language. In order to provide a consistent scenario without the need of a comprehensive explanation of its domain, the examples in this Chapter follow the Starship MTheory used in [2], which was based on the Star Trek television series2 . The explanations and examples presented here assume no previous familiarity with the particulars of the Star Trek series. 1 2

UnBBayes-MEBN is available at http://sourceforge.net/projects/unbbayes. Star Trek and related marks are registered trademarks of Paramount Pictures.

20

R.N. Carvalho et al.

2.2.1

MEBN Fragments

An MFrag represents a conditional probabilistic distribution of the resident RVs instances given the values of its parent instances in the graph. This distribution can only be applied if the domain context restrictions are satisﬁed. One of the advantages of using MEBN instead of BN is that the number of RV instances is not limited and does not need to be previously known. RVs are instantiated dynamically and represent domain entities. Directed arrows going from parent to child variables in the graph present relations between these entities. Thus, an MFrag represents the conditional probability distribution of its RV instances given the values of their parents, as long as the context nodes are satisﬁed. In the Star TrekT M series, the U.S.S. Enterprise starship is equipped with various sensors that provide information the Commander needs to keep it out of danger. The MFrag in Figure 2.1 represents the level of danger a given starship is exposed to.

Fig. 2.1. An example of a Starship MFrag

This MFrag has seven nodes: four context nodes, two input nodes and one resident node. The context nodes are Boolean variables that represent conditions that have to be satisﬁed so that the probabilistic distribution of the resident nodes applies. Their possible values are: True (the condition is satisﬁed), False (the condition is not satisﬁed), and Absurd (a condition expression does not make sense). For instance, if s in the node IsOwnStarship(s) is replaced by the Enterprise identiﬁer, the context node value will be true. If it is replaced by any other starship identiﬁer then the context node value will be false. If any other thing that is not a starship replaces the argument, the context node value will be Absurd. Input nodes are variables that inﬂuence the probabilistic distribution of its child resident nodes, but their distributions are deﬁned within their own MFrags. In other words, in a complete MTheory, every input node must be a resident node in another MFrag, where its probabilistic distribution will be deﬁned. Resident

2 A GUI Tool for Plausible Reasoning in the Semantic Web Using MEBN

21

Fig. 2.2. A code example to produce a CPT

nodes have the local probabilistic distributions deﬁned in that MFrag, including the probabilistic dependence on its parent values (that can be input or resident nodes). A node can have a list of arguments in parenthesis, which are replaced by unique identiﬁers of domain entities when the net is instantiated. By convention, these identiﬁers begin with an exclamation point and two distinct entities cannot have the same unique identiﬁer. In Figure 2.1, node DangerToSelf(s,t ) has two arguments: s and t. In this domain, s must be a starship and t a discrete point in time. The instance DangerToSelf(!Enterprise,!T0 ) evaluates the level of danger the Enterprise is exposed to during the time instance T0. Also, the values of nodes states are not shown in the MFrag graph. This is because an MFrag is just a template, in other words, it does not represent individuals RVs, but a class of RVs. The values of its states appear only when the MFrag is instantiated. For instance, to discover the probabilistic distribution of an instance of the variable DangerToSelf(s,t ), it is ﬁrst necessary to ﬁnd all instances of HarmPotential(st,t ) and OpSpec(st ) for which the context nodes are satisﬁed. A pseudo code to produce the probabilistic distributions of the DangerToSelf(s,t ) variable is showed in the Figure 2.2. This pseudo code gives rise to the local distribution for the danger to which a starship is submitted given all the starships that inﬂuence its level of danger. This distribution takes into consideration that the number of starships is not previously known. This situation cannot be represented using BNs. However, for greater ﬂexibility, the speciﬁcation of PR-OWL CPTs and formulas was not completely deﬁned in [2]. Section 2.4 introduces the UnBBayes-MEBN approach to deal with this problem. 2.2.2

MEBN Theories

A MEBN theory, or MTheory for short, is a collection of MFrags that satisﬁes consistency constraints ensuring the existence of a unique joint probabilistic distribution (JPD) over the RVs mentioned in the theory. This JPD is speciﬁed

22

R.N. Carvalho et al.

by the local distribution and pattern of each MFrag, which collectively form a generative MTheory. This information can be modeled in a knowledge base using a combination of experts and Bayesian network learning. In a complete MTheory, each RV has just one home MFrag where its local distribution is deﬁned. See [6] for a more information on MTheories. An MTheory assigns probabilities to sets of worlds. This is done in a way that ensures that the set of worlds consistent with the logical content of the MTheory has probability 100%. Each random variable instance maps a possible world to the value of the random variable in that world. In statistics, random variables are deﬁned as functions mapping a sample space to an outcome set. For MEBN random variable instances, the sample space is the set of possible worlds. For example, ZoneNature(!Z0 ) maps a possible world to the nature of the zone labeled !Z0 in that world. The probability that !Z0 is a deep space zone is the total probability of the set of possible worlds for which ZoneNature(!Z0 ) has value DeepSpace. In any given possible world, the generic class random variable ZoneNature(z ) maps its argument to the nature of the zone whose identiﬁer was substituted for the argument z. Thus, the sample space for the class random variable ZoneNature(z ) is the set of unique identiﬁers that can be substituted for the argument z. Information about statistical regularities among zones is represented by the local distributions of the MFrags whose arguments are zones. As more information is obtained about which possible world might be the actual world, the probabilities of all related properties of the world must be adjusted in a logically coherent manner. This is accomplished by adding ﬁndings to an MTheory to represent the new information, and then using Bayesian conditioning to update the probability distribution represented by the revised MTheory. For example, suppose the system receives conﬁrmed information that at least one enemy starship is navigating in !Z0. This information means that worlds in which ZoneEShips(!Z0 ) (which means the number of enemy starships in the given zone) has value Zero are no longer possible. In classical logic, this new information makes no diﬀerence to the inferences one can draw about ZoneNature(!Z0 ). All four values (One, Two, Three, and MoreThanThree starships) were possible before that new information arrived (i.e. there’s at least one enemy starship in !Z0 for sure), and all four values remain possible. The situation is diﬀerent in a probabilistic logic. To revise the current probabilities, it is necessary to ﬁrst assign probability zero to the set of worlds in which !Z0 contains no enemy starships. Then, the probabilities of the remaining worlds should be divided by the prior probability that ZoneEShips(!Z0 ) had a value other than Zero. This ensures that the set of worlds consistent with the new knowledge has probability 100%. These operations can be accomplished in a computationally eﬃcient manner using SSBN construction algorithm. Figure 2.3 depicts the Star Trek MTheory used by Costa [2]. In a simpliﬁed way, the main task of this MTheory is to model the problem of detecting Romulan starships (here considered as hostile by the United Federation of Planets) and assessing the level of danger they bring to the own starship, the Enterprise.

2 A GUI Tool for Plausible Reasoning in the Semantic Web Using MEBN

23

All other starships are considered either friendly or neutral. Starship detection is performed by the Enterprise’s suite of sensors, which can correctly detect and discriminate starships with an accuracy of 95%. However, Romulan starships may be in “cloak mode”, which makes them invisible to the Enterprise’s sensors. Even for the most current sensor technology, the only hint of a nearby starship in cloak mode is a slight magnetic disturbance caused by the enormous amount of energy required for cloaking. The Enterprise has a magnetic disturbance sensor, but it is very hard to distinguish background magnetic disturbance from that generated by a nearby starship in cloak mode. There are four basic entities in this MTheory: Starship, Zone, TimeStep, and SensorReport. According to the Treknology Encyclopedia3 , Starship is the designation for a large type of space vessel with warp drive. A starship typically consists of more than one deck and has separate departments such as the bridge, engineering or sickbay. In this model, this word is used to designate any space vessel. A zone can be either a Deep Space, a Planetary System, or the Boundary of a Black Hole. It is assumed that a OwnStarship, when in operation, has 80% chance of being traveling in a Deep Space Zone, 15% in a Planetary System and 5% in the Boundary of a Black Hole. In this model, black hole boundaries are preferred places for ambushes from attacking starships with cloaking devices, since the high magnetic turbulance generated in those zones makes it very hard to even the most advanced sensors to distinguish it from the magnetic disturbance created by a cloaking device. TimeStep is a special class that is used to model dynamic nodes, and there are quite a few in this domain. Finally, starship detection is performed by the Enterprise’s suite of sensors, which can correctly detect and discriminate starships with an accuracy of 95%. The product of those sensors are individuals of the class SensorReport. MEBN logic allows multiple, equivalent ways of portraying the same knowledge. Nevertheless, Costa [2] encourages the use of object oriented approach sought with the Star Trek MTheory, which used the concept of an entity cluster deﬁned also in [2]. Deﬁnition 1. An entity cluster is a group of MFrags within a generative MTheory having the following characteristics: 1. in any MFrag contained in the entity cluster, there is an ordinary variable, called the subject argument of the MFrag, such that any nonconstant random variable in the MFrag has the subject argument as one of its arguments. 2. the context constraints of each MFrag in the entity cluster specify the type of the subject argument. This type is called the subject type of the MFrag. 3. the subject types of all MFrags in the entity cluster are the same. Having this in mind, it is possible to map this MTheory in four clusters. The TimeStep MFrag for the TimeStep cluster. The Zone MFrag for the Zone cluster. The Starship Existence, Starship Data, Starship, Danger To Others, and 3

http://www.ex-astris-scientia.org/treknology1.htm

24

R.N. Carvalho et al.

Danger To Self MFrags for the Starship cluster. Finally, the SR Data and Sensor Report MFrags for the SensorReport cluster. Using this modeling approach makes it easier to keep MEBN logic’s ﬂexibility to display the same information in diﬀerent MFrag conﬁgurations. The other MFrags, IsA and Entity Type, are used for deﬁning type safety and it is better explained in [2]. The TimeStep cluster is used to make the use of recursion with this entity possible and it is explained in details also in [2]. In the Zone cluster there are the following resident nodes: ZoneNature(z ) has been explained above, when the type Zone was described. ZoneEShips(z ) establishes the relationship between a given zone and the likelihood of having enemy starships within OwnStarship’s sensor range. In other words, it is the probable number of enemy ships into sensor range assumed to ﬁnd in a given zone. This means it is considered that exists a prior probability of ﬁnding an enemy starship given the nature of the zone in which OwnStarship is navigating through. In this model, the inﬁnitely possible number of starships was restrained to only ﬁve states. That is, it is assumed that it is unlikely to ﬁnd four or more hostile ships in that area, so most of the probability distribution mass for this node will be restricted to the states None, One, Two, and Three, while the remaining probability will be restricted to the aggregating state MoreThanThree. ZoneFShips(z ) establishes the relationship between a given zone and the likelihood of having friendly starships within OwnStarship’s sensor range. Following the very same rationale of node ZoneEShips(z ), it is assumed that there is a prior probability in the number of friendly or neutral starships to appear into OwnStarship’s sensor range given the nature of the zone it is navigating. ZoneMD(z,t ) assesses the value of the magnetic disturbance in Zone z at the current TimeStep t. This value is inﬂuenced by the MD in the previous TimeStep (tprev ), the fact of whether there is or there is not a starship in cloak mode nearby, and the nature of the space zone in which the starship is located. The input node t=!T0 is used to “anchor” the time recursion. In the Starship cluster there are the following resident nodes: IsOwnStarship(st ) identify if st is the OwnStarship, in this case the Enterprise. StarshipZone(st ) identify the Zone the starship st is navigating into. Exists(st ) is the probatility of existence for Starships. It is a useful way of conveying hypothetical instances of a Starship since there is a prior probability of ﬁnding enemy or friendly starships depending on where OwnStarship is navigating, these parameters will also inﬂuence the prior probability of existence. Thus ZoneEShips(z ) and ZoneFShips(z ), where Zone z is the same Zone the Starship st is at, are parents of Exists(st ). OpSpec(st ) conveys the information on what species is operating a given starship. Its distribution is derived from the number of Friendly and Enemy Starships in the vicinity.

2 A GUI Tool for Plausible Reasoning in the Semantic Web Using MEBN

25

StarshipClass(st ) assesses what is the class of the starship represented by st. It is inﬂuenced by the kind of species that is operating the starship and the very own existence of the starship itself (as deﬁned in the context node Exists(st )). There is a vast literature of classes and subclasses of starships for each species (e.g. see http://techspecs.acalltoduty.com). However, for this simple model it is used a general taxonomy that aggregates the starships in ﬁve diﬀerent classes (WarBird, Cruiser, Explorer, Frigate and Freighter ). CloakMode(st ) is a boolean variable that deﬁnes whether the starship in question is in cloak mode. In this model, it is assumed that only Romulan and Klingon starships can be in cloak mode, since the Federation still does not have such technology. DistFromOwn(st,t ) assesses the distance from a starship st to OwnStarship at TimeStep t. This distance is measured according to weapon’s ranges, since its main purpose is to assess the ability to any given starship to harm OwnStarship. HarmPotential(st,t ) assesses the potential of starship st to harm OwnStarship at current TimeStep t. It is based on the starship weapons’ range (based on its class) and its distance from OwnStarship. It is important to note that here it is not being assessed the intention to harm, but only the ability to do so. Therefore, even friendly starships can have HarmPotential with value true (e.g. provide that they are within their respective weapons range). DangerToSelf(s,t ) assesses the level of danger to which OwnStarship s is exposed at a given time t. Basically, this danger level will be a funcion of the ability of a starship st to harm OwnStarship and of the intention of whoever is operating starship st to harm OwnStarship, the latter being implied from the knowledge of what species is operating starship st. DangerToOthers(s,t ) conveys the ability of OwnStarship s to inﬂict danger to another starship st at TimeStep t. It is based on OwnStarship’s weapons (implicitly considered in the probability distribution) and its distance from starship st. In the SensorReport cluster there are the following resident nodes: Subject(sr ) has as its possible values all the unique identiﬁers of the entities that can be the subject of the sensor report being represented by the variable sr. In this model, sensor reports can refer to starships (real or hypothetical), in which case the node will assume the unique identier of that starship as its value, or it can refer to nothing (i.e. a spurious report), in which case it will assume the unique identiﬁer of a spurious report as its value (e.g. OSpurious). SRClass(sr,t ) conveys the result of a sensor report sr regarding to the class of a given starship at current TimeStep t. SRDistance(sr,t ) conveys the result of a sensor report sr regarding to the distance of a given starship to OwnStarship at current TimeStep t. Each of these eleven MFrags represents the probability information about a group of their respective random variables. Collectively, the group implicitly

26

R.N. Carvalho et al.

Fig. 2.3. The Starship MTheory

expresses a JPD over truth-values of sets of FOL sentences. That is, probability distributions are speciﬁed locally over small groups of hypotheses and composed into globally consistent probability distributions over sets of hypotheses. MEBN theories extend ordinary Bayesian networks to provide an inner structure for random variables. Random variables in MEBN theories take arguments that refer to entities in the domain of application. As an example from the Sensor Report MFrag, the predicate StarshipClass(st ) might represent the class of the starship designated by the variable st. To refer to the class of the starship instance labeled !Avenger, one would ﬁll in a value for st to obtain an instance StarshipClass(!Avenger ) of the StarshipClass(st ) random variable. Remind that, in MEBN syntax, an exclamation mark starts a string when it is used as an instance label. Captain Picard, the Commander of the Enterprise starship, has more than an academic interest in the danger from nearby starships. He must make decisions with life and death consequences. Multi-Entity Decision Graphs (MEDGs, or “medges”) extend MEBN logic to support decision making under uncertainty. MEDGs are related to MEBNs in the same way inﬂuence diagrams are related to Bayesian networks. A MEDG can be applied to any problem that involves optimal choice from a set of alternatives subject to given constraints. When a decision MFrag (i.e. one that has decision and utility nodes) is added to a generative MTheory such as the one portrayed in Figure 2.3, the result is a MEDG. As an example, Figure 2.4 depicts a decision MFrag representing Captain Picard’s choice of which defensive action to take. The decision node DefenseAction(s) represents the set of defensive actions available to the Captain (in this case, to ﬁre the starship’s weapons, to retreat, or to do nothing). The value nodes capture Picard’s objectives, which in this case are to protect the Enterprise while also avoiding harm to innocent people as a consequence of his defensive actions. Both objectives depend on Picard’s decision, while ProtectSelf(s) is inﬂuenced by the perceived danger to Enterprise and ProtectOthers(s) is depends on the level of danger to other starships in the vicinity.

2 A GUI Tool for Plausible Reasoning in the Semantic Web Using MEBN

27

Fig. 2.4. The Star Trek Decision MFrag

The model described here is clearly an oversimpliﬁcation of any “real” scenario a Captain would face. Its purpose is to convey the core idea of extending MEBN logic to support decision-making. Indeed, a more common situation is to have multiple, mutually inﬂuencing, often conﬂicting factors that together form a very complex decision problem, and require trading oﬀ diﬀerent attributes of value. For example, a decision to attack would mean that little power would be left for the defense shields; a retreat would require aborting a very important mission. MEDGs provide the necessary foundation to address all the above issues. Readers familiar with inﬂuence diagrams will appreciate that the main concepts required for a ﬁrst-order extension of decision theory are all present in Figure 2.4. In other words, MEDGs have the same core functionality and characteristics of common MFrags. Thus, the utility table in Survivability(s) refers to the entity whose unique identiﬁer substitutes for the variable s, which according to the context nodes should be our own starship (Enterprise in this case). Likewise, the states of input node DangerToSelf(s,t ) and the decision options listed in DefenseAction(s) should also refer to the same entity. Of course, this confers to MEDGs the expressive power of MEBN models, which includes the ability to use this same decision MFrag to model the decision process of the Captain of another starship. Notice that a MEDG Theory should also comply with the same consistency rules of standard MTheories, along with additional rules required for inﬂuence diagrams (e.g., value nodes are deterministic and must be leaf nodes or have only value nodes as children). 2.2.3

MEBN Inference Algorithm

The MTheory depicted in Figure 2.3 is a generative MTheory, which provides prior knowledge that can be updated upon receipt of evidence represented as ﬁnding MFrags. We now describe the process used to obtain posterior knowledge from a generative MTheory and a set of ﬁndings. In a BN model, assessing the impact of new evidence involves conditioning on the values of evidence nodes and applying a belief propagation algorithm. When the algorithm ﬁnishes, beliefs of all nodes, including the node(s) of interest, reﬂect the impact of all evidence entered thus far. This process of entering

28

R.N. Carvalho et al.

evidence, updating beliefs, and inspecting the posterior beliefs of one or more nodes of interest is called belief propagation. Usually, the belief propagation process is carrying on answering probabilistic queries. MEBN inference works in a similar way (after all, MEBN is a Bayesian logic), but following a more complex yet more ﬂexible process. Whereas BNs are static models that must be changed whenever the situation changes (e.g. number of starships, time recursion, etc.), an MTheory implicitly represents an inﬁnity of possible scenarios. In other words, the MTheory represented in Figure 2.3 is a model that can be used for as many starships as wanted, and for as many time steps that are necessary to get the conclusions needed. That said, the obvious question is how to perform queries within such a model. In [6], Laskey proposed a SSBN construction algorithm that uses an initial generative MTheory, a ﬁnding set (which conveys particular information about the situation), and a target set (which indicates the nodes of interest to the query being made). UnBBayes-MEBN uses a variation of that algorithm, which is explained in Section 2.4.2. For comparison, let’s suppose there is a situation in which four starships are within the Enterprise’s range. In this particular case, a BN can be used to represent the situation at hand, which means the model would be “hardwired” to a known number (four) of starships, and any other number would require a different model. A standard Bayesian inference algorithm applied to that model would involve entering the available information about these four starships (i.e., the four sensor reports), propagating the beliefs, and obtaining posterior probabilities for the hypotheses of interest (e.g., the four Starship Type nodes). Similarly, MEBN inference begins when a query is posed to assess the degree of belief in a target random variable given a set of evidence random variables. It is started with a generative MTheory, add a set of ﬁnding MFrags representing problem-speciﬁc information, and specify the target nodes for the query. The ﬁrst step in MEBN inference is to construct the SSBN, which can be seen as an ordinary Bayesian network constructed by creating and combining instances of the MFrags in the generative MTheory. Next, a standard Bayesian network inference algorithm is applied. Finally, the answer to the query is obtained by inspecting the posterior probabilities of the target nodes. The UnBBayes-MEBN algorithm does not handle decision graphs at this stage of development. Thus, the illustration presented in the following lines extends the algorithm for purposes of demonstrating how the MEDG Theory formed by adding the MFrag in Figure 2.4 to the MTheory in Figure 2.3 can be used to support the Captain’s decision. In this example, the ﬁnding MFrags convey information that there are ﬁve starships (!ST0 through !ST4 ) and that the ﬁrst is Enterprise itself. For the sake of illustration, let’s assume that the ﬁnding set also includes data regarding the nature of the space zone Enterprise is currently located (!Z0 ), its magnetic disturbance for the ﬁrst time step (!T0 ), and sensor reports (!SR1 to !SR4 ) for starships for the ﬁrst two time steps. Figure 2.5 shows the situation-speciﬁc Bayesian network for such query. To construct that SSBN, the initial step is to create instances of the random

2 A GUI Tool for Plausible Reasoning in the Semantic Web Using MEBN

29

Fig. 2.5. SSBN for the Star Trek MTheory with Four Starships within Range

variables in the MTheory and the random variables for which there are ﬁndings. The random variables of interest are DangerLevel(!ST0 ) and DefenseAction(!ST0 ). The ﬁnding random variables are the eight SRDistance nodes (2 time steps for each of four starships) and the two ZoneMD reports (one for each time step). Although each ﬁnding MFrag contains two nodes, the random variable on which there is a ﬁnding and a node indicating the value to which it is set, only the ﬁrst of these is included in the situation-speciﬁc Bayesian network, and declared as evidence that its value is equal to the observed value indicated in the ﬁnding MFrag. Evidence nodes are shown with bold borders. UnBBayes-MEBN algorithm starts retrieving and instantiating random variables of the MTheory and its ﬁndings database. That way, when random variables are created, they represent known background information, observed evidence, and queries of interest to the decision maker. If there are any random variables with undeﬁned distributions, then the algorithm proceeds by using its respective default distribution. The process continues until there are no remaining random variables having either undeﬁned distributions or unknown values. The result, if this process terminates, is the SSBN or, in this example, a situation-speciﬁc decision graph (SSDG). Mahoney and Laskey [9] deﬁne a SSBN as a minimal Bayesian network sufﬁcient to compute the response to a query. A SSBN may contain any number of instances of each random variable, depending on the number of entities and their interrelationships. The SSDG in Figure 2.5 is the result of applying this process to the MEDG Theory obtained with the aggregation of Figure 2.3 and Figure 2.4 with the ﬁnding and target set deﬁned above.

2.3 An Overview of the PR-OWL Language The usual workaround for representing probabilities in deterministic languages like OWL is to show probability information as annotations. This means that

30

R.N. Carvalho et al.

numerical information is stored as text strings. Because this solution does not convey the structural features of a probabilistic domain theory, it is no more than a palliative. This is no a minor shortcoming. Researchers have stressed the importance of structural information in probabilistic models (see [12]). For instance, Shafer ([13], pages 5-9) stated that probability is more about structure than it is about numbers. A major concept behind PR-OWL is that of probabilistic ontologies, which goes beyond simply annotating ontologies with probabilities to provide a means of expressing all relevant uncertainties about the entities and relationships that exist in a domain in a logically coherent manner. This not only provides a consistent representation of uncertain knowledge that can be reused by diﬀerent probabilistic systems, but also allows applications to perform plausible reasoning with that knowledge, in an eﬃcient way. PR-OWL uses the following deﬁnition of a probabilistic ontology [2]: Deﬁnition 2. A probabilistic ontology is an explicit, formal knowledge representation that expresses knowledge about a domain of application. This includes: 1. 2. 3. 4. 5. 6. 7.

types of entities existing in the domain; properties of those entities; relationships among entities; processes and events that happen with those entities; statistical regularities that characterize the domain; inconclusive, ambiguous, incomplete, unreliable, and dissonant knowledge; uncertainty about all the above forms of knowledge;

where the term entity refers to any concept (real or ﬁctitious, concrete or abstract) that can be described and reasoned about within the domain of application. Probabilistic ontologies are used for the purpose of comprehensively describing knowledge about a domain and the uncertainty associated with that knowledge in a principled, structured, and sharable way. PR-OWL was developed as an extension enabling OWL ontologies to represent complex Bayesian probabilistic models in a way that is ﬂexible enough to be used by diverse Bayesian probabilistic tools based on diﬀerent probabilistic technologies (e.g. PRMs, BNs, etc.). More speciﬁcally, PR-OWL is an upper ontology (i.e. an ontology that represents fundamental concepts that cross disciplines and applications) for probabilistic systems. PR-OWL is expressive enough to represent even the most complex probabilistic models. It consists of a set of classes, subclasses and properties that collectively form a framework for building probabilistic ontologies. Currently, the ﬁrst step toward building a probabilistic ontology as deﬁned above is to import the PR-OWL ontology into an ontology editor (e.g. OntoEdit, Protege, Swoop, etc.) and start constructing the domain-speciﬁc concepts, using the PR-OWL deﬁnitions to represent uncertainty about their attributes and relationships. Using this procedure, a knowledge engineer is not only able to build a coherent generative MTheory and other probabilistic ontology elements, but also make it compatible with other ontologies that use PR-OWL concepts. However, building MFrags this way is a manual, error prone, and tedious process

2 A GUI Tool for Plausible Reasoning in the Semantic Web Using MEBN

31

that requires deep knowledge of the logic and of the data structures of PR-OWL in order to avoid errors or inconsistencies. UnBBayes-MEBN changes all that by providing a GUI-based editing process for building probabilistic ontologies based on the PR-OWL upper ontology on probabilistic models [1]. The major advantages of using PR-OWL are its ﬂexibility and representational power, both inherited from the fact that the language is based on MEBN, a full integration of First-Order Logic and probability that merges the expressiveness of the former with the inferential power of the latter. UnBBayes-MEBN leverages that power with a built-in MEBN reasoner that implements both the SSBN creation process and its respective evaluation. The next section provides an overall view of the current state of that tool. The prospective reader can ﬁnd additional details on PR-OWL at http://www.pr-owl.org. PR-OWL was proposed as an extension to the OWL language based on MEBN, which can express a probabilistic distribution under any axiomatic FOL theory model. As a consequence, there are no guaranties that the reasoning with PR-OWL ontology will be eﬃcient or even decidable [2]. PR-OWL was built to be interoperable with non-probabilistic ontologies. Since PR-OWL adds new deﬁnitions to current OWL while retaining backward compatibility with its base language, OWL-built legacy ontologies will be able to interoperate with newly developed probabilistic ontologies. However, the ontology’s probabilistic deﬁnitions have to form a valid MTheory. OWL has three diﬀerent versions with increasing expressive power designed for speciﬁc communities of developers and users. The less expressive is OWL Lite, which has a limited set of simple restrictions. More expressiveness is found in OWL DL, which is based in Descriptive Logic and aims to maximize expressiveness while maintaining its completeness (all conclusions are computable) and decidability (all computations end in a ﬁnite time). It has all OWL constructions, but they have to be used under certain restrictions. The most powerful version, OWL Full, was built for users that want the most expressiveness possible. As a consequence, there are no guaranties of computability. Following the same reasoning, a PR-OWL Lite version could be created as suggested in [2] with some restrictions. Figure 2.6 shows the main concepts involved in deﬁning an MTheory in PR-OWL. In the diagram, ellipses represent general classes while arrows represent the main relationships between these classes. A probabilistic ontology (PO) has to

Fig. 2.6. Concepts of the PR-OWL language

32

R.N. Carvalho et al.

Fig. 2.7. PR-OWL ontology elements

have at least one individual of class MTheory, which is basically a label linking a group of MFrags that collectively form a valid MTheory. In actual PR-OLW syntax, that link is expressed via the object property hasMFrag (which is the inverse of object property isMFragIn). Individuals of class MFrag are comprised of nodes, which can be resident, input, or context nodes (not shown in the picture). Each individual of class Node is a random variable RV and thus has a mutually comprehensive, collectively exhaustive set of possible states. In PR-OWL, the object property hasPossibleValues links each node with its possible states, which are individuals of class Entity. Finally, random variables (represented by the class Nodes in PR-OWL) have unconditional or conditional probability distributions, which are represented by class ProbabilityDistribution and linked to its respective nodes via the object property hasProbDist. Figure 2.7 depicts the main elements of the PR-OWL language, its subclasses, and the secondary elements necessary for representing an MTheory. The relations necessary to express the complex structure of MEBN probabilistic models using the OWL syntax are also depicted. Building MFrags and all its elements into a PO is a hard, tiring and error prone process. It demands deep knowledge of PR-OWL’s syntax, semantics and data structure. UnBBayes-MEBN GUI was built to address this problem, providing a visual tool for building MEBN models. Section 2.4 shows the advantages of the GUI. Another important feature is the ability to save and open models created by the UnBBayes-MEBN GUI in PR-OWL format, with backwards compatibility to OWL through the use of the Protege API. Protege is an ontology editor and a ﬂexible and conﬁgurable framework for building knowledgebase tools and applications that has been developed by the Stanford Medical Informatics.

2 A GUI Tool for Plausible Reasoning in the Semantic Web Using MEBN

33

2.4 UnBBayes-MEBN The current lack of a software tool makes the task of creating a PO using the PROWL language very diﬃcult. Basically, the easiest way of building PR-OWL ontologies is via a graphical ontology editor such as Protege. In this case, PR-OWL deﬁnitions have to be imported into Protege (from http://www.pr-owl.org/prowl.owl), making the task of building a PO a bit easier because it is not necessary to remember all information and OWL tags that should be ﬁlled. However, the input of information is not intuitive and the user has to know all technical terms as hasPossibleValues, isNodeFrom, hasParents, etc (Figure 2.8). Many of these terms could be omitted and ﬁlled automatic by a software application such as UnBBayes-MEBN, designed to enforce the consistency of a MEBN model. UnBBayes-MEBN was planned to allow building a PO in an intuitive way and not having to rely on a deep knowledge about the PR-OWL speciﬁcation. In addition, a click in the “R” icon and another click anywhere in the edition panel will create a resident node. This is shown in Figure 2.9.

Fig. 2.8. Node speciﬁcation with Protege

After that, clicking in the “+” button allows to ﬁll a name and to add the states of the node. All the remaining tasks required by PR-OWL syntax (e.g. ﬁlling the terms as isResidentNodeIn, etc.) are automatically done. Figure 2.10 shows how UnBBayes allows a more adequate and better visualization of the MTheory and MFrags being created, as well as their nodes. In short, it is not

34

R.N. Carvalho et al.

Fig. 2.9. Node speciﬁcation with UnBBayes-MEBN

Fig. 2.10. General view of the MTheory and MFrag in UnBBayes

2 A GUI Tool for Plausible Reasoning in the Semantic Web Using MEBN

35

diﬃcult to perceive the advantages of building POs with the GUI implemented in UnBBayes-MEBN. Implementing a complex logic such as MEBN while focusing on the usability requirements of an (probabilistic) ontology editor requires making trade-oﬀs between performance, decidability, expressivity, and ease of use. In other words, the complexity of the logic and the fact that it is still in development imply that any implementation has to include alternative algorithms and optimizations to make a working, feasible tool. UnBBayes-MEBN is no exception to this rule, and many of the design decisions were based on the above-cited constraints. 2.4.1

The API and Data Structure

Probabilistic ontologies in UnBBayes-MEBN, or UnBBayes for short, are saved with the PR-OWL format, which is an extension of OWL format. UnBBayes uses the Java open source Protege application programming interface (API) for dwelling with OWL ﬁles. Protege allows for the edition of ontologies in two ways: using Protege-Frames editor and using Protege-OWL editor. The ontologies can be saved in various formats, including RDF, OWL, and XML Schema. UnBBayes provides support for MEBN input/output operations using the Protege-OWL API, which is based on the class JenaOWLModel. Protege uses the Jena API for various tasks, in particular for parsing OWL/RDF ﬁles. We will now describe the implementation components used for saving and opening MTheories. Package unbbayes.io.mebn has the classes that are needed for saving and opening MEBN models in the supported format. Interface MebnIO is the interface that the classes implementing input/output features for a given format must provide. These classes are LoadMebn (loads a MEBN from a valid ﬁle) and SaveMebn (saves a MEBN in a valid ﬁle). Class PrOwlIO implements the MebnIO interface to save and load ﬁles in the PR-OWL format. It uses the Protege-OWL API to achieve this. This API allows an MTheory created using the UnBBayes GUI to be saved, while also keeping compatibility with any editions made in Protege. An MTheory saved in Protege can be opened in UnBBayes and vice-versa. This compatibility is important because it ensures that ﬁles created in UnBBayes can be opened and edited in any OWL-compliant application such as Protege (although these applications will not be able to understand the ontology’s probabilistic characteristics). In addition, ontologies that have already been deﬁned using Protege can be extended to the PR-OWL format in a quick and direct way. All that is needed is to open the OWL ﬁle in UnBBayes and create a MTheory for this ontology and save the result. In order to account for the complexity of PR-OWL/MEBN elements, an object-oriented (OO) MEBN data structure was designed. Some elements were not implemented as their use was either not relevant or outside the scope of this initial stage of the tool (such as the exemplar variables described in [6]).

36

R.N. Carvalho et al.

Fig. 2.11. Main MEBN elements design

Initially, extra elements were included into the design to account for the algorithms’ inner logical tasks such as the ﬁnding MFrag, ﬁnding nodes and others. However, design choices such as the use of PowerLoom rendered much of these extra elements unnecessary, facilitating the implementation. That was possible because PowerLoom was capable of handling all ﬁnding information, as it will be explained later on. The main MEBN elements, as implemented in the tool, can be seen in Figure 2.11. The MultiEntityBayesianNetwork (MEBN for short) depicted in the ﬁgure accounts for Network that can be edited in UnBBayes, and it is a collection of MFrag (when it is composed by the more general deﬁnition of resident and input nodes), or DomainMFrag (when it is composed by the more speciﬁc definition of resident and input nodes). There are three types of nodes deﬁned: context, input, and resident. The input and resident nodes have more speciﬁc deﬁnitions: generative and domain respectively. The ResidentNode is the only one that has a table associated, through the interface ITabledVariable. Another useful information is that the InputNode is an input instance of a BuiltInRV or a ResidentNode. Also part of UnBBayes-MEBN design is the built-in implementation of type and ordered type for recursion. In Figure 2.12, every entity has a type and the implementation enforces a type-safe policy. As for recursion, the design solution involves the ObjectEntityInstanceOrdereable, also in Figure 2.12, which ensures that a certain type, TimeStep for instance, has a total order for every instance

2 A GUI Tool for Plausible Reasoning in the Semantic Web Using MEBN

37

Fig. 2.12. Main MEBN Entity and Type design

created, enabling the assessment of the previous and next instances. Using this approach, implementing recursion was mostly reduced to allowing a node to be set as recursive and setting the stop condition by deﬁning the last instance in the recursion for a given entity. This approach is currently being considered to be included as part of the PR-OWL speciﬁcation. MEBN, as a ﬁrst-order Bayesian logic, poses the implementation challenge of how to evaluate FOL sentences. A diﬃcult task on its own, the design option involved searching for an open-source API to deal with it. The choice was PowerLoom , a Knowledge Representation and Reasoning (KR&R) tool that is a highly expressive, logic-based KR&R system with multiple built-in deductive reasoning capabilities including a query processor, a description classiﬁer, and a context mechanism. It was developed at the University of Southern California as a successor to the successful Loom KR&R system, with Hans Chalupsy as the project leader. PowerLoom is now distributed in three options of open source licensing terms: GPL, LGPL, and Mozilla. Using PowerLoom in the UnBBayes-MEBN implementation forced the team to design the KnowledgeBase and PowerLoomKB to build, load, and save a knowledge base (KB) from a MTheory. It can load and save ﬁnding and generative information through its modules within the environment. Through the environment, the KBFacade and PowerLoomFacade make it easier within the MEBN implementation to verify whether an entity exists, to evaluate formulas, to search for ﬁndings, and to obtain other useful information about the KB. This design is shown in Figure 2.13. 2.4.2

SSBN Construction Algorithm

UnBBayes-MEBN also includes a knowledge base (KB) to store domain information. When a query is submitted, the KB is searched for information to answer the query. If the available information does not suﬃce, then the KB and the

38

R.N. Carvalho et al.

Fig. 2.13. Knowledge Base and PowerLoom design

generative MTheory are used to construct a BN to answer the query. This process is called SSBN construction. In the current implementation, a query consists of a single random variable (RV) instance, which is not allowed to have any evidence below it. The following procedure takes a node name and a list of entity instances as arguments. It is called initially with the query node and its arguments. PROCEDURE SSBN-CNSTR(NODE, ENTITY-LIST): 1. For the RV instance NODE(ENTITY-LIST), search for evidence in the KB. If there is a ﬁnding for this given entry, ﬁnish. 2. Search for the resident node that has the name NODE and get its MFrag. Once NODE(OV-LIST) is found, verify if the type of ENTITY-LIST is the same as OV-LIST (where OV-LIST is the list of ordinary variable arguments for NODE in its home MFrag). 3. Verify in the KB which context nodes refer to the OVs in OV-LIST, replacing each OV by the appropriate instance in ENTITY-LIST. If any context variable is false, mark the MFrag to use the default distribution. 4. If the truth-value of the context node in (iii) is not determined, make it a parent of NODE. 5. For each parent of NODE, identify any instance of the parent that can be constructed by replacing the OVs by the known entities (contained in the query or KB), and has not yet been added to the SSBN. For each such

2 A GUI Tool for Plausible Reasoning in the Semantic Web Using MEBN

39

parent instance, call procedure SSBN-CNSTR for the parent node and its arguments. 6. Create the NODE’s CPT. 7. Finish. This algorithm is easily enhanced to allow multiple query nodes and evidence below query nodes. These enhancements are currently under development. A few performance issues had to be considered in the implementation of UnBBayesMEBN. Depending on the complexity of the domain, the algorithm may reach a context node that cannot be immediately evaluated. This happens when all ordinary variables in the parents set of a resident random variable term do not appear in the resident term itself. In this case, there may be an arbitrary, possibly inﬁnite number of instances of a parent for any given instance of the child. For example, in the Starship MFrag depicted in Figures 2.1 and Figure 2.2, if the zone where a starship is located is uncertain, the number of enemies and friends (ZoneEStarships(z ) and ZoneFStarships(z )) in any zone it might be located is relevant to the distribution of the OpSpec(st ) random variable. If time step t has previous time steps, then more than one distance (DistanceFromOwn(st,tprev )) must be evaluated, which makes the distance measured in all time steps relevant to the distribution of the DistFromOwn(st,tprev ) random variable in time t. Thus, any number of instances of the ZoneEShips(z ), ZoneFShips(z ), and DistFromOwn(st,tprev ) random variables might be relevant to the distributions of the OpSpec(st ) and DistFromOwn(st,tprev ) random variables in time step t. In this case, the local distribution for a random variable must specify how to combine inﬂuences from all relevant instances of its parents. However, especially in complex formulas this may have a strong impact in the performance of the algorithm, so the designed solution involves asking the user for more information. In the current implementation, if one does not provide such information the algorithm will just halt.

Fig. 2.14. SSBN construction for HarmPotential(!ST4,!T3 ) query - Step 1

40

R.N. Carvalho et al.

Fig. 2.15. SSBN construction for HarmPotential(!ST4,!T3 ) query - Step 2

Fig. 2.16. SSBN construction for HarmPotential(!ST4,!T3 ) query - Step 4

Another design option was to restrict memory usage in a way that a possible memory overload triggers a warning to the user and stops the algorithm. In step (iii), a design optimization over the general SSBN algorithm in [6], only the necessary context nodes for a given MFrag are evaluated, in contrast with the original solution of revising all the context nodes for that MFrag. Although the implementation addressed other optimization issues, for the sake of conciseness only the most relevant are listed here. To better understand the SSBN construction procedure shown above, an step by step example will be shown. Suppose the following query is entered in UnBBayes, after some entity instances and ﬁndings are deﬁned: HarmPotential(!ST4,!T3 ). Once the procedure starts (see Figure 2.14), it will try to look for a ﬁnding for this query and it will not ﬁnd. Therefore, it will search for the MFrag with the given node ﬁnding the Starship MFrag. As the their types match, the procedure continues. It veriﬁes wich context nodes to evaluate, which gives: IsA(Starship,st ), IsA(TimeStep,t ), Exists(st ), and ˜IsOwnStarship(st ). They all

2 A GUI Tool for Plausible Reasoning in the Semantic Web Using MEBN

41

Fig. 2.17. SSBN construction for HarmPotential(!ST4,!T3 ) query - Step 5

Fig. 2.18. SSBN construction for HarmPotential(!ST4,!T3 ) query - Step 9

return true, so the node HarmPotential(!ST4,!T3 ) is created. Now it is time to call the same procedure for its parents. It ﬁrst ﬁnds the DistFromOwn(st,t ) node, for the entities !ST4 and !T3. It goes through the same steps, but with one diﬀerence (see Figure 2.15), when it is about to call the same procedure for its parent, it realizes that it is a recursive deﬁnition, as its parent has the same name and arguments type and it also has a orderable argument, TimeStep. Therefore, it will keep calling the procedure for its parent but with the previous TimeStep, !T2 in this case, until it reaches the last TimeStep or a ﬁnding. In Figure 2.16 it ﬁnally reaches a ﬁnding for the TimeStep !T0. As it has no more parents, the node DistFromOwn(!ST4,!T1 ) can generate its CPT. In Figure 2.17 it is possible to see the resulting CPT for DistFromOwn (!ST4,!T1 ) and this node is ﬁnally added as a parent of DistFromOwn(!ST4,!T2 ). As DistFromOwn(!ST4,!T2 ) has no more parents, it can also generate its CPT.

42

R.N. Carvalho et al.

Fig. 2.19. SSBN construction for HarmPotential(!ST4,!T3 ) query - Step 18

Fig. 2.20. SSBN generated for the HarmPotential(!ST4,!T3 ) query

Some similar steps later (see Figure 2.18), it veriﬁes that the next parent is deﬁned in another MFrag, so it has to change its context for the new MFrag and call the procedure for ZoneFShips(!Z2 ), as this Zone was the one that made the context node z =StarshipZone(!ST4 ) return true. Once all nodes have generated their CPTs and the query node HarmPotential(!ST4,!T3 ) has no more parents to evaluate (see Figure 2.19), it is then ready to generate its CPT and ﬁnish the SSBN construction returning the generated BN, or the situation-speciﬁc Bayesian network, for a normal BN belief propagation (see Figure 2.20). 2.4.3

Modiﬁcations to the Original PR-OWL

PR-OWL was designed as a general formalism for building POs, without a focus on implementing an actual tool such as UnBBayes-MEBN. Thus, some

2 A GUI Tool for Plausible Reasoning in the Semantic Web Using MEBN

43

contributions from this work were actually introduced into the revised PR-OWL speciﬁcation. One example is the the possibility of deﬁning the instances of a given entity as possible state values for a resident node. In the Starship Data MFrag, for instance, the node StarshipZone(st ) has all instances of the Zone entity as its possible state values. This contribution was accepted and incorporated in PR-OWL version 1.03. As a consequence, it was necessary to deﬁne a standard for the CPT associated to this given resident node. As the number of instances for the entity is unknown and the pseudocode can not be used in this case, the adopted solution was to deﬁne its CPT as a uniform distribution. Another example is the grammar deﬁned for writing formulas to dynamically build CPTs, which addresses a key aspect of reasoning process: the generation of the combined probability tables (CPT) for nodes with dynamically deﬁned distributions. More speciﬁcally, when a query is posed to the system, it triggers the algorithm for building the Situation Speciﬁc Bayesian Network (SSBN) that will answer the query by instantiating all random variables that can add information towards its solution. In this process, some nodes may have an unknown number of parents, so instead of a static, previously deﬁned CPT there will be a formula for dynamically generating the CPT given the number of parents in that speciﬁc situation. Although the original work on the PR-OWL language does not contain a rigid speciﬁcation for representing formulas that perform the dynamic deﬁnition of probability distributions of an instantiated MFrag’s nodes, it presents pseudo code ([2], page 64) that was used as a basis for specifying conditional probabilistic tables (CPT) in UnBBayes-MEBN. The implementation of this CPT generator is a key aspect for the SSBN construction, as many real world models include nodes with no previously deﬁned number of parents. In UnBBayesMEBN, a grammar and a complete new compiler, featuring a lexical, syntactic, and semantic analyzer were designed from the ground up. During the SSBN construction algorithm, they act by evaluating dynamic probability distribution formulas and then building the CPT for the respective node. Figure 2.21 shows a sample of the grammar’s pseudo-code, which can be understood as a sequence of probability assignment blocks conditioned by if-else clauses. The clause else has a special purpose: it not only declares non-speciﬁed probabilities, but also establishes the default distribution (i.e. the one used when a context node is not satisﬁed). Evaluation of the pseudo-code is performed via syntactical analysis by a recursive descent parser, which is a top-down technique in which each production rule is implemented via a procedure. Commands and additional data structures were also added to the grammar as a means to allow for a semantic analysis of the code and for the generation of intermediate code. The latter is composed of speciﬁc blocks that are evaluated once all random variable conditioning cases are known, and is responsible for performing the ﬁnal SSBN CPT generation. As an example of an element of the grammar, the production rule varsetname declares how the pseudo-code references a given set of parent nodes. Parent node instances can be divided into subsets that have similar predominant arguments. To better

44

R.N. Carvalho et al.

Fig. 2.21. Grammar used for dynamically generating a CPT

understand the concept, suppose that all parent nodes that have arguments st and z as their respective predominant arguments form a subset of parent nodes. Then, a hypothetical condition “if any st.z” would be applied only to members of that subset. Currently, non-predominant arguments (weak arguments) are the recursive ordinary variables (e.g. ordinary variable t in DangerToSelf(s,t )). Another contribution to PR-OWL was the inclusion of information on global exclusivity, useful in situations when only one ﬁnding is allowed for a speciﬁc node in a given state. For instance, in Starship Data MFrag of Figure 2.1, the IsOwnStarship(st ) has the state True as possible for just one starship st. That is, the state True has globally exclusive with respect to the RV IsOwnStarship(st ). Global Exclusivity was accepted as a contribution by the PR-OWL team, and was inserted in PR-OWL version 1.05 (www.pr-owl.org).

2.5 Conclusions This Chapter presents a proposal for a GUI that facilitates the creation of probabilistic ontologies built on MFrags and MTheories. In addition, it also presents a MEBN CPT formula editor that can deal with previously unknown number of nodes. Besides that, it presents the algorithm implemented for the SSBN generation with an detailed step by step example. This work is currently being implemented in the UnBBayes software, which is alpha phase at the time of this writing and is able to create, load, and save MEBN models in the PR-OWL ﬁle format. This research represents a contribution to the SW community and,

2 A GUI Tool for Plausible Reasoning in the Semantic Web Using MEBN

45

more speciﬁcally, to the current work of the URW3-XG Incubator Group, created by the W3C to better deﬁne the challenge of reasoning with and representing uncertain information available through the World Wide Web and related technologies.

References 1. Carvalho, R.N., Santos, L.L., Ladeira, M., Costa, P.C.G.: A GUI Tool for Plausible Reasoning in the Semantic Web using MEBN. In: Proceedings of the Seventh International Conference on Intelligent Systems Design and Applications, Rio de Janeiro, Brazil, October 2007, pp. 381–386. IEEE Computer Society Press, Los Alamitos (2007) 2. Costa, P.C.G.: Bayesian Semantics for the Semantic Web. PhD thesis, Department of Systems Engineering and Operational Research, George Mason University (2005) 3. Costa, P.C.G., Laskey, K.B.: PR-OWL: A Framework for Probabilistic Ontologies. In: Proceedings of the Fourth International Conference on Formal Ontology in Information Systems, Baltimore, USA (2006) 4. Costa, P.C.G., Laskey, K.B., Laskey, K.J.: PR-OWL: A Bayesian Ontology Language for the Semantic Web. In: Proceedings of the ISWC Workshop on Uncertainty Reasoning for the Semantic Web, Galway, Ireland (2005) 5. Heﬂin, J.: OWL Web Ontology Language - Use Cases and Requirements (W3C Recommendation) (2004), www.w3.org/TR/2004/REC-webont-req-20040210 6. Laskey, K.B.: MEBN: A Language for First-Order Bayesian Knowledge Bases. Artiﬁcial Intelligence 172(2–3), 172–225 (2007) 7. Laskey, K.B., Costa, P.C.G.: Of Klingons and Starships: Bayesian Logic for the 23rd Century. In: Proceedings of the Twenty-ﬁrst Conference Uncertainty in Artiﬁcial Intelligence (UAI 2005), Edinburgh, Scotland, pp. 346–353 (2005) 8. Laskey, K.J., Laskey, K.B., Costa P.C.G.: Uncertainty Reasoning for the World Wide Web Incubator Group Charter (W3C Incubator Activity) (2007), www.w3.org/2005/Incubator/urw3/charter 9. Mahoney, S.M., Laskey, K.B.: Constructing Situation Speciﬁc Networks. In: Proceedings of the Fourteenth Conference on Uncertainty in Artiﬁcial Intelligence (UAI 1998), University of Wisconsin Business School, Madison, WI, USA, pp. 370–378 (July 1998) 10. Patel-Schneider, P.F., Hayes, P., Horrocks, I.: OWL Web Ontology Language Semantics and Abstract Syntax (W3C Recommendation) (2004), www.w3.org/TR/owl-semantics/ 11. Pearl, J.: Probabilistic Reasoning in Intelligent Systems: Networks of Plausible Inference. Morgan Kaufmann Publishers, San Mateo (1988) 12. Schum, D.A.: Evidential Foundations of Probabilistic Reasoning. Wiley Interscience, Hoboken (1994) 13. Shafer, G.: The Construction of Probability Arguments. Boston University Law Review 66(3–4), 799–823 (1986) 14. Toﬄer, A.: The Third Wave. Morrow, New York (1980)

3 Multiobjective Optimization and Rule Learning: Subselection Algorithm or Meta-heuristic Algorithm? Celso Y. Ishida1 , Aurora Pozo2 , Elizabeth Goldbarg3, and Marco Goldbarg3 1

2

3

PhD, Numerical Methods in Engeneering Program - Federal University of Paraná UFPR P.O. Box 19081, 81531-970 - Curitiba, Brazil [email protected] Computer Science Department - Federal University of Paraná - UFPR [email protected] Computer Science Department - Federal University of Rio Grande do Norte - UFRN {beth,marco}@dinamp.ufrn.br

Summary. A previous work explores a Multi-Objective Subset Selection algorithm, denominated the Pareto Front Elite, to induce classiﬁers. These classiﬁers are composed by a set of rules selected following Pareto dominance concepts and forming unordered classiﬁers. These rules are previously created by an association rule algorithm. The performance of the classiﬁers induced were compared with other well known rule induction algorithms using the area under the ROC curve. The area under the ROC curve (AUC) is considered a relevant criterion to deal with imbalanced data, misclassiﬁcation costs and noisy data. The results show that the Pareto Front Elite algorithm is comparable to the best known techniques. In this paper we explore multi-objective meta-heuristic approach to create rules and to build the Pareto Front using the sensitivity and speciﬁcity criteria, the chosen Metaheuristic is a Greedy Randomized Adaptive Search Procedure (GRASP) with path-relinking. We perform an experimental study to compare the two algorithms: one based on a complete set of rules, and the other based on Metaheuristic Approach. In this study we analyze the classiﬁcation results, through the AUC criterion, and the Pareto Front coverage produced by each algorithm.

3.1 Introduction A signiﬁcant need exists for techniques and tools with the ability to intelligently assist humans in analyzing very large collections of data in search of useful knowledge, in this sense, the area of data mining has received special attention. Research and development in data mining evolve in several directions, which are not necessarily divergent. Rules are one of the most used forms to represent knowledge in data mining context. This is because of their simplicity, intuitive aspect, modularity and can be obtained directly from a data set [14]. Therefore, rules induction has been established as a fundamental component of many data mining systems. Furthermore, it was the ﬁrst machine learning technique to become part of successful commercial data mining applications [5]. N. Nedjah et al. (Eds.): Innovative Applications in Data Mining, SCI 169, pp. 47–70. c Springer-Verlag Berlin Heidelberg 2009 springerlink.com

48

C.Y. Ishida et al.

Our work deals with learning classiﬁcation rules, or rule induction. In the rule induction process, the algorithm receives a data set as input, in supervised mode. This data set must contain examples with the descriptor attributes and the class values. This attributes could have discrete values, i.e. gender (male, female), or numerical values, i.e. age (can varies in a range of 0 to 120). The algorithm looks to the examples and searches to identify patterns. These patterns are going to be represented in rules. A rule is a pair or if antecedent then consequent. The antecedent means the restrictions in the values of the attributes for one example be classiﬁed as the consequent, the class value. After the end of the learning process, new examples that arrive can be classiﬁed by the learned rules. In this sense, the task of learning a rule is a combinatorial optimization problem where the complexity grows with the number of descriptor attributes and the number of examples on the database. One problem in classiﬁcation rules is how to deﬁne a performance metric. Numerous measures are used for performance evaluation of rules, but all of them are derived from the contingency table [30]. A contingency table for an arbitrary rule with antecedent B and consequent H is showed at Table 3.1. In Table 3.1, B denotes the set of instances for which the body of the rule is true, and B denotes its complement (the set of instances for which the body is false); similarly for H and H. HB then denotes H ∩ B, HB denotes H ∩ B, and so on. Table 3.1. A contingency table B B H n(HB) n(HB) n(H) H n(HB) n(HB) n(H) n(B) n(B) N

In Table 3.1, n(X) denotes the cardinality of the set X, e.g., n(HB) is the number of instances for which H is false and B is true (i.e., the number of instances erroneously covered by the rule). N denotes the total number of instances in the data set. In the task of learning classiﬁcation rules, the goal is to create rules that together have good performance for classiﬁcation. The great majority of the methods search to optimize the performance of the classiﬁcation by optimizing the precision in training set [14]. But, there are some problems with the precision metric for rules induction. For example, precision metric does not consider the presence of unbalanced classes. The solution for these cases is to use probabilistic classiﬁers [14]. A frequent measure used to evaluate the performance of a probabilistic classiﬁer is the area under the ROC curve (Receiver Operating Characteristic). A ROC curve is a graph that relates the false positive rate (FP) (axis-x) and the true positive rate (TP) (axis-y) of a classiﬁer [35]. A ROC curve can be obtained from a unique rule, a partial classiﬁer or a complete classiﬁer. Recently, a number of researchers have expressed the performance of their

3 Multiobjective Optimization and Rule Learning

49

classiﬁcation algorithms based on the area under the ROC curve (AUC - Area Under Curve) [34] [16] [38] [40]. The goal is to maximize the AUC. Traditional rule induction systems often uses a greedy algorithm. This is a covered approach where a search procedure is iteratively executed. In this search, on each iteration, the algorithm ﬁnds the best rule and removes all the examples covered by this rule from the data set. Then, the process is repeated with the remaining examples [33]. The process continues until all the examples are covered or some stop criterion is reached. In this way, on each iteration, a new rule is found. However, this approach has major problems. The removal of the examples from the data set at each new discovered rule causes the over-specialization of the rules after some iterations. This means that each rule covers few examples. Besides that, the classiﬁer composed by the learned rules is an ordered list where the interpretation of one rule depends on the precedent rules. In the last years, an hybrid approach is gaining attention. This approach consists of an associative classiﬁer [32] [42]. An associative classiﬁer is composed of all the rules generated by an association rules algorithm in which the consequent attribute is the class attribute. For generating the complete set of rules, the main attractive of an associative classiﬁer against the rule learning algorithms is the possibility to obtain potentially good rules that could be discarded due to the use of heuristic methods [12]. However, this easiness usually comes with the cost of generating a large set of rules. Some recent works have suggested an additional step to remove the redundant and irrelevant rules in order to have a more compact associative classiﬁer [26]. Some works based on this approach are: the ROCCER algorithm [33], the GARSS [2] and the PFE [23]. In a previous work we introduced the PFE, a rule subset selection algorithm. This algorithm allows conceiving a novel approach where the properties of the rules can be expressed with diﬀerent objectives. The algorithm selects these rules from a large set of rules (in the same way the Apriori algorithm [1] does). In this way, the algorithm allows creating classiﬁers composed by rules with speciﬁc properties that explore the concepts of Pareto dominance. These rules can be used as an unordered classiﬁer. The rules are more intuitive and easier to understand because they can be interpreted independently one of the other. Furthermore, the quality of the learned rules is not aﬀected during the learning process because the data set is not modiﬁed. An algorithm called Pareto Front Elite (PFE) was implemented with this approach using the speciﬁcity and the sensitivity criteria, deﬁned in equations 3.1 and 3.2. Experiments were made comparing the PFE with other well known algorithms using the Area Under the ROC curve metric (AUC), and the results were very encouraging. sensitivity =

n(HB) n(H)

(3.1)

n(HB) (3.2) n(H) This previous work motivated the project and implementation of new algorithms based on meta-heuristic approaches [22]. The research with those algorithms specif icity =

50

C.Y. Ishida et al.

investigated an open question: Can these rules be created by a multi-objective meta-heuristic strategy? Two Metaheuristic algorithms were implemented and compared: a Greedy Randomized Adaptive Search Procedure with path-relinking (GRASP-PR), and Multiple Objective Particle Swarm (MOPS). The comparison showed better results for the GRASP-PR algorithm. In this work, we explore more deeply the quality of the solutions obtained with the GRASP-PR algorithm and compared them with the PFE. The data provided by this comparison will support conclusions about the beneﬁts of utilizing a metaheuristic approach or an approach based on a complete set of rules. Both techniques are applied to diﬀerent data sets and the results are compared to traditional systems. Furthermore, the analysis of the coverage of the Pareto Front obtained by both systems is reported. This paper is organized as follows. The next section outlines related works. Section 3.3 describes the Pareto Front Elite algorithm and Section 3.4 explains the GRASP-PR Rule Learning Algorithm. The Section 3.5 provides some basic concepts from multi-objective optimization and outlines the methodology used to compare multi-objective algorithms. Section 3.6 presents results of a computational experiment where the performance of the GRASP-PR and the PFE are compared. Finally, Section 3.7 concludes the work.

3.2 Related Work Several works were proposed to ﬁnd a set of rules for the classiﬁcation task. This section describes two lines of works more related to our paper: The subselection algorithms and the multiobjective meta-heuristic works. The works [31], [32], [42], [33] and [2] use the association rules algorithm to generate the rules. Due to the huge number of rules generated by those algorithms, [26] presents the AprioriC where the redundant rules are removed. However, the AprioriC still generates a large set of rules. The ROCCER [33] algorithm constructs a convex hull in ROC space. The ROCCER uses the Apriori algorithm with Minimum support and conﬁdence to generate a large set of rules. The idea is to insert a rule that leads to a point outside the current ROC convex hull. In addition to the need of the Apriori algorithm, the insertion and the removal of rules imply on backtracking, a time consuming procedure for the ROCCER algorithm. Using the set of rules as the classiﬁcation model, the Pareto Front Elite algorithm [23] has the same goal as the algorithms detailed above, i.e., to maximize the AUC. One association rule algorithm is executed to generate rules with support greater than a Minimum support parameter or Conﬁdence greater than a Minimum conﬁdence parameter. From the rule set, the algorithm uses the sensitivity and the speciﬁcity criteria in a post-process method to select the Pareto Front. If the minimum parameters are set to very low values then this algorithm can be considered as an all rules approach, i.e., all the rules with high sensitivity or speciﬁcity values are in the classiﬁer induced by the Pareto Front Elite algorithm.

3 Multiobjective Optimization and Rule Learning

51

Recently, increasing interest has emerged in applying the concept of Paretooptimality to machine learning inspired by the successful developments in evolutionary multiobjective optimization. These researches include multiobjective feature selection, multiobjective model selection in training multilayer perceptrons, radial-basis-function networks, support vector machines, decision trees and intelligent systems [25]. In the literature, few works deal with multiobjective evolutionary algorithms for rule learning between them [21], [10] and [20]. The ﬁrst work focuses on the rule selection phase. It presents a genetic-based multiobjective rule selection algorithm to ﬁnd a smaller rule subset with higher accuracy than extracting the rule sets by heuristic approaches. The algorithm has the objective to maximize the accuracy, and to minimize the number of rules. In [20], multiobjective association rules generation and selection with NSGA-II (Non-Dominated Sorting Genetic Algorithm) are discussed. In [9], a multiobjective optimization evolutionary algorithm with Pareto concepts is used to discover interesting classiﬁcation rules for a target class. It presents an implementation of NSGA with positive conﬁdence and sensitivity as objectives. This work is extended in [10] using multiobjective metaheuristics to produce sets of interesting classiﬁcation rules. A measure of dissimilarity of rules was introduced to promote diversity on the population. Our work focuses on the induction of an unordered classiﬁer where the rule generation and the rule selection occur at the same time. As a result, there is no need of selecting the best rules for the classiﬁer after the generation process. The multiobjective metaheuristic approaches aim at inducing classiﬁers composed by rules with speciﬁc properties. In order to achieve this goal, we choose a Greedy Randomized Adaptive Search Procedure with path-relinking (GRASP-PR) using the sensitivity and speciﬁcity as objectives.

3.3 Pareto Front Elite Pareto Front Elite (PFE) algorithm was introduced at [24]. The main goal is to select a set of non-ordered rules for the maximization of the AUC criterion. In order to accomplish this goal, ﬁrst a set of rules are generated by an association rule algorithm (as Apriori). From this rule set, Pareto Front Elite algorithm uses two important measures of rules: sensitivity and speciﬁcity as a simple post process method to select the Pareto Front. The main steps detailed in the Algorithm 3.1 are: the execution of the association rule algorithm to generate rules (EliteRules) and the selection of the Pareto Front (paretoF ront). These steps are executed for each class. The execution of the association rule algorithm, like Apriori algorithm, need the speciﬁcation of two parameters: Minimum support and Minimum conﬁdence. These parameters are needed to control the number of generated rules. The output of this step is a set of rules, which we call Elite rules, these rules are the best rules according to support or conﬁdence. In the second step, paretoF ront at Algorithm 3.1, the Elite rules are examined and selected according to sensitivity and speciﬁcity criteria to be part of the Pareto Front.

52

C.Y. Ishida et al.

Algorithm 3.1. Pareto Front Elite Input: class - Head of the rules (positive / negative) Output: paretoF ront - Pareto Front General Steps: EliteRules = Apriori( class, M inimumSupport, M inimumConf idence ) paretoF ront = {rulej ∈ EliteRules / ¬∃rulei , > Sensitivity(rulej ) Sensitivity(rulei ) Specif icity(rulej )}

and

Specif icity(rulei )

>

The initial results of Pareto Front Elite Algorithm were presented at paper [24]. In [23], empirical studies were carried out to verify the inﬂuence of the Apriori Minimum support and Minimum conﬁdence parameters in Pareto Front Elite algorithm. These parameters were used to generate a smaller or larger set of rules. The aim was to decrease the Minimum support and Minimum conﬁdence Parameters expecting the increase of the mean AUC values. Here, these results are detailed to allow a full comparison with the meta-heuristic algorithm. 3.3.1

Experimental Results for Pareto Front Elite Algorithm

This subsection explains the methodology used to evaluated the PFE algorithm. First the chosen parameters for the PFE algorithm are described and after, the data set and compared algorithms are presented. Finally, the results are shown followed by a discussion. Parameters Three sets of parameters for the PFE algorithm were chosen, these parameters were calculated based on all rules with one attribute at each data set. Let M eanSupport be the mean of the support values, StdDevSupport be the standard deviation of the support values, M eanConf idence be the mean of the conﬁdence values and StdDevConf idence be the standard deviation of the conﬁdence values. The ﬁrst set of parameters is the same used at [24]: the Minimum support parameter is set to M inSupport1 (equation 3.3) and the Minimum conﬁdence parameter is set to M inConf idence1 (equation 3.4). These are the highest values for the parameters. The second set of parameters is the same used at [23] deﬁned by: M inSupport2 (equation 3.5) and M inConf idence2 (equation 3.6). These values increase the generated rule set, still keeping still good values of support or conﬁdence. Finally, Minimum support and Minimum conﬁdence are set to zero (equations 3.7 and 3.8), yielding the generation of all rules.

3 Multiobjective Optimization and Rule Learning

M inSupport1 = M eanSupport + StdDevSupport

53

(3.3)

M inConf idence1 = M eanConf idence + StdDevConf idence M inSupport2 = M eanSupport/2

(3.4) (3.5)

M inConf idence2 = M eanConf idence/2 M inSupport3 = 0

(3.6) (3.7)

M inConf idence3 = 0

(3.8)

Methodology To compare the Pareto Front Elite algorithm with the related works, it is used the results reported by [33]. The experiments of the Pareto Front Elite algorithm are done with the same methodology and paired data sets from the UCL Machine Learning Repository [11]. Table 3.2 presents the description of data sets used in the experiments. It shows the number of attribute (’Att’ column), number of examples (’Exa’ column) and number of classes (’Cl’ column) of the sixteen data sets. We run the experiments using 10-fold stratiﬁed cross-validation and for all inducers the same training and test ﬁles are employed. All the comparisons are done using t-test with 95% conﬁdence level. The rules obtained were used as a classiﬁer using a weighted voted classiﬁcation process based on the conﬁdence [14]. The results of the AUC values are estimated using the trapezoidal integer method. Table 3.2. Description of data sets used in the experiments # Data set Att Exa Cl # Data set 1 2 3 4 5 6 7 8

breast bupa ecoli ﬂag german glass haberman heart

10 7 8 29 21 10 4 14

683 345 336 174 1000 214 306 270

2 2 8 6 2 6 2 2

9 10 11 12 13 14 15 16

Att Exa

ionosphere 34 kr-vs-kp 37 lettera 17 new-thyroid 6 nursery 9 pima 9 satimage 37 vehicle 19

351 3196 20000 215 12960 768 6435 846

Cl 2 2 26 3 5 2 6 4

Other systems The Pareto Front Elite’s results were compared with the algorithms ROCCER [33], C4.5 [37], CN2 [4], Ripper [6] and Slipper [7]. The ROCCER algorithm were executed with 50% of Minimum Conﬁdence parameter and Minimum support equal to 1/3 of minority class. The C4.5 [36] uses information gain as quality measure to build a decision tree and a post-pruning step based on error reduction. Both the pruned (C45) and non-pruned (C45NP) versions are evaluated. The ﬁrst version of CN2 [4] induces a decision list using the entropy as a search heuristic. The CN2 algorithm was executed in the ordered (CN2OR)

54

C.Y. Ishida et al.

and unordered (CN2) versions. Ripper is a learning algorithm of decision rules list [6]. It has the parameter error-based pruning and an MDL-based heuristic for determining how many rules should be learned. The Slipper [7] use the weighted set-covering to improve the Ripper. These results, mean AUC values and standard deviations between brackets, are shown at Table 3.3. Table 3.3. Mean AUC (Other Systems) # ROCCER C45 C45NP CN2 CN2OR Ripper Slipper 1 98.63(1.88) 97.76(1.51) 98.39(1.3) 99.26(0.81) 99.13(0.92) 98.72(1.38) 99.24(0.57) 2 65.3(7.93) 62.14(9.91)57.44(11.92) 62.74(8.85) 62.21(8.11) 69.1(7.78) 59.84(6.44) 390.31(11.56) 50(0) 90.06(7.75) 90.17(6.9)85.15(11.38)61.86(25.49)74.78(15.94) 461.83(24.14) 50(0)68.68(17.22)53.22(24.12)42.78(24.43)45.28(14.93) 52.35(7.44) 5 72.08(6.02) 71.43(5.89) 67.71(4.12) 75.25(5.38) 70.9(4.7)64.02(13.62) 71.32(6.2) 679.45(12.98) 50(0) 81.5(12.65) 73.74(15.4)79.64(13.24) 49.75(0.79) 50(2.36) 766.41(11.54) 55.84(6.14)64.33(13.58) 59.83(9.87)59.28(10.13) 57.45(3.85) 50.4(11.14) 8 85.78(8.43) 84.81(6.57) 81.11(7.91) 83.61(6.89) 82.25(6.59) 84.89(7.68) 84.03(6.36) 9 94.18(4.49) 86.09(9.97) 90.91(6.03) 96.23(2.97) 92.18(7.54) 92.06(5.94) 93.95(6.82) 10 99.35(0.36) 99.85(0.2) 99.86(0.2) 99.85(0.16) 99.91(0.17) 99.85(0.21) 99.91(0.09) 11 96.08(0.52) 95.49(1.96) 99.33(0.46) 99.34(0.28) 99.44(0.63) 97.27(1.86) 98.82(0.44) 12 98.4(1.7)87.85(10.43) 97.5(3.39) 99.14(1.19) 98.43(2.58) 94.95(9.94) 99.12(1.25) 13 97.85(0.44) 99.42(0.14) 99.74(0.13) 100(0) 99.99(0.01) 99.43(0.26) 94.4(1.59) 14 70.68(5.09) 72.07(4.42) 72.6(6.5) 70.96(4.62) 71.97(5.44) 68.07(9.46) 70.02(5.97) 15 89.39(2.38) 90.15(1.7) 91.31(1.32) 91.48(1.45) 91.48(0.9) 86.83(3.94) 89.06(1.98) 16 96.42(1.47) 94.76(3) 96.99(1.44) 97.38(2.05) 96.49(2.41) 95.01(2.22) 93.99(3.13) Avg 85.13 77.98 84.84 84.51 83.20 79.03 80.08

Observe that, for some data sets it was not possible to run PFE due to computer resources limitations, the number of rules created by the association rules algorithm is too large for these support and conﬁdence parameters. The limitation is due to the association rules algorithm. Results The results of the PFE algorithm, mean AUC values and standard deviations between brackets, using the three set of parameters are shown at Table 3.4. The ﬁrst analysis consider the results with the ﬁrst set of parameters. We observe that PFE algorithm has good AUC performance for the majority of the data sets. The results show that Pareto Front Elite algorithm obtains a similar performance to the ROCCER algorithm. The algorithm is one time better than the ROCCER algorithm (data set #16), and, it is worse at just data set #13. The best performances are against C45, Ripper and Slipper algorithms, where the Pareto Front Elite has, respectively, better AUC values at 5, 4 and 4 data sets. There is a balance with C45NP and CN2OR algorithms. However, the CN2 has the best performance against the Pareto algorithm. Analyzing the data sets, at most of them, the PFE algorithm has similar values to others algorithms.

3 Multiobjective Optimization and Rule Learning

55

Table 3.4. Mean AUC PF Elite # PF Elite (3.5)(3.6) PF Elite (3.3)(3.4) PF Elite (3.7)(3.8) 1 99.36(0.87) 99.33(0.95) 99.39(0.88) 2 65.37(10.37) 63.83(10.29) 65.40(8.77) 3 90.94(7.16) 91.55(4.78) 92.22(6.29) 5 73.78(6.52) 73.51(6.49) 75.86(6.71) 6 78.07(12.35) 77.07(14.36) 80.86(13.48) 7 61.92(14.51) 66.75(6.03) 59.65(11.65) 8 88.78(6.82) 88.56(7.00) 88.61(6.38) 11 95.53(1.4) 95.46(1.41) — 12 97.27(3.98) 97.41(4.00) 97.76(3.06) 13 98.89(0.36) 98.82(0.43) 100.00(0.00) 14 70.98(5.88) 70.41(3.87) 72.01(4.94) 16 95.95(1.94) 93.72(2.8) —Avg 84.74 84.70 —-

But, at data set #11(lettera), #13(nursery) and #16(vehicle), the Pareto Front Elite algorithm has bad results against most of the algorithms. The next analysis considers the results with the second set of parameters where exists the increase of the number of rules for the construction of the Pareto Front with the decrease of the Minimum parameters. The results with the parameter set to M inSupport2 (equation 3.5) and M inConf idence2 (equation 3.6) are presented in the second column of Table 3.4 and were also depicted in [23]. We observe that Pareto Front Elite algorithm keeps the good AUC performance for the majority of the data sets. The exception is for the data set #16, where there is a signiﬁcant improvement. With the ﬁrst set of parameters, the Pareto Front Elite algorithm looses for four algorithms, now, the algorithm has AUC values similar to the others. Comparing with other algorithms, there is an improvement for Pareto Front Elite algorithm. The PFE algorithm has similar or better values at most of the data sets. The exceptions are for the CN2 and CN2OR algorithms. In summary, with the second parameter setting, the Pareto Front Elite algorithm still selects a set of rules with high AUC values. There is an improvement of the AUC for the data set #16 and, in no cases, the parameters M inSupport2 and M inConf idence2 are statistically worse than M inSupport1 and M inConf idence1. To better understand the eﬀect of these parameters in the Pareto Front Elite algorithm, the third column of Table 3.4 presents the results of setting to zero the values for the Minimum support and Minimum conﬁdence parameters, i.e., all the rules are created. We examined the eﬀect in ten data sets, the ones that was possible to generate all the rules due to our resource limitations.

56

C.Y. Ishida et al.

As expected, the lower parameter generates classiﬁers with more rules but with the AUC higher. Despite of the better values, only for the data set #13, the AUC value is statistically diﬀerent. It can be observed the overﬁtting at the data set #7. The third column of the line correspondent to data set #7 shows the best AUC value (66.75) found for this data set. It is the result of the algorithm execution with the greatest values for the Minimum parameters. Lower values for the parameters produce an increasing on the number of generated rules. As a result, the decrease of the AUC to 61.92 and 59.65 is observed in the second and fourth columns, respectively, of Table 3.4. To obtain good results, the Pareto Front Elite algorithm obtains a large set of rules compared to other algorithms. The good results are due to the quality of the rules in the Pareto Front. The third line at Table 3.5 presents the mean support of the rules for the highest values for the Minimum support parameter and Minimum conﬁdence parameter. The rules from all Pareto Front for these parameters have the mean equal to 15.26. With the exception of the ROCCER algorithm, all the other algorithms have signiﬁcant lower mean support. Analyzing the two results of the Pareto Front Elite algorithm (second and third lines), a lower value with the parameter M inSupport2 (equation 3.5) and M inConf idence2 (equation 3.6) is expected. Even with a lower mean support, the result is statistically similar to the ROCCER algorithm and better than other algorithms. The column WRAcc of Table 3.5 has the Weighted Relative Accuracy measure that indicates the rule importance. There is no statistical diﬀerence between the two results of Pareto Front Elite Algorithm, ROCCER and Ripper algorithms, being all others worse. The drawback for the Pareto Front Elite Algorithm is the execution at the data sets with many attributes. Even with the minimum parameters of support and conﬁdence, the Apriori algorithm is not able to generate the rules for the large data sets according to the number of attributes (data set #4, #9, #10 and #15). This emphasizes the necessity of the substitution of the deterministic algorithm of the process. Table 3.5. Mean Support and Weighted Relative Accuracy Algorithm Support (%) PF Elite (Eq. 3.5;Eq. 3.6) 13.19 ( 4.15 ) PF Elite (Eq. 3.3;Eq. 3.4) 15.26 ( 5.20 ) ROCCER 13.67 ( 13.89 ) C4.5 3.73 ( 6.01 ) C4.5NP 1.19 ( 1.06 ) CN2 3.9 ( 2.52 ) CN2OR 3.1 ( 2.18 ) Ripper 5.96 ( 5.34 ) Slipper 1.92 ( 1.58 )

WRAcc 0.0261 ( 0.020 ) 0.0282 ( 0.022) 0.0355 ( 0.018 ) 0.0094 ( 0.013 ) 0.003 ( 0.003 ) 0.011 ( 0.009 ) 0.0085 ( 0.007 ) 0.0184 ( 0.012 ) 0.006 ( 0.006 )

3 Multiobjective Optimization and Rule Learning

57

3.4 GRASP Greedy Randomized Adaptive Search Procedure (GRASP) [15] is a metaheuristic approach initially created for Combinatorial Optimization Problems. GRASP is a multi-start process where each iteration consists of two phases: the construction of initial solutions and the local search. The construction phase builds a feasible solution using a greedy randomized strategy. From these initial solutions the local search phase explores the neighborhood until reaching a local optimum. At each iteration of the construction phase, a feasible solution is constructed one element at a time. A candidate element is a piece that can be incorporated into the partial solution without destroying the feasibility. The candidate elements are ordered using a function that measures the beneﬁt of the element for the solution. The best n-candidate elements are selected to be at the Restricted Candidate List (RCL), where n is a parameter. The GRASP randomly chooses one element from the RCL to incorporate into the solution. This way, diﬀerent solutions are obtained at each iteration. After then, the RCL is updated and the functions are evaluated again reﬂecting the beneﬁts of the inclusion of the chosen element. The ﬁrst phase does not assure generating local optima with respect to simple neighborhood deﬁnitions [17]. For that, local search is applied in order to improve each solution constructed in the ﬁrst phase. In the local search algorithm, the current solution is iteratively replaced by a better neighbor solution. The process terminates when no better solution is found in the neighborhood of the current solution. The solutions generated by the GRASP algorithm can be signiﬁcantly improved with path-relinking algorithm. The use of path-relinking algorithm within a GRASP algorithm was ﬁrst proposed by Laguna and Marti [29]. The pathrelinking can be applied as an intensiﬁcation strategy, as a post-optimization step to all pairs of best solutions resulting of the local search [39]. Path-relinking was proposed as a search strategy that explores trajectories connecting elite solutions obtained by tabu search. It oﬀers a wide range of mechanisms for creating productive combinations of input solutions [18]. Given two solutions, their common elements are kept constant and the space of solutions spanned by these elements is searched with the aim of ﬁnding a better solution. At next, we present our algorithm that uses the concepts presented here to improve the results at the classiﬁcation task. 3.4.1

The GRASP-PR Rule Learning Algorithm

We use the meta-heuristics presented above to propose the GRASP-PR Rule Learning Algorithm. The algorithm was introduced at [22] and its pseudo-code is presented at Algorithm 3.2. The algorithm has two phases: The GRASP phase with the construction of rules (ConstructGreedyRandomizedSolution) and the local search (LocalSearch) to improve the solutions. This phase creates a list of rules, called Elite Rules (EliteRules), that will be processed further in the

58

C.Y. Ishida et al.

Algorithm 3.2. GRASP-PR Rule Learning Algorithm InputInstance() procedure grasp() boolean T P bias = true // true=TP rate as bias; false=TN rate as bias for i = to maxLoop Solution = ConstructGreedyRandomizedSolution( T P bias, percentageElements) localBest = LocalSearch(Solution, T P bias) UpdateSolution( localBest, EliteRules) T P bias = not T P bias endfor EliteRules = ﬁlterGeneric( EliteRules) return( EliteRules) end grasp non_dom = InsertParetoFront ( EliteRules ) procedure path-relinking( EliteRules,non_dom ) foreach rule1 ∈ EliteRules foreach rule2 ∈ EliteRules path (rule1 , rule2 , non_dom ) endforeach endforeach return(non_dom) procedure path(rule1 , rule2 , non_dom) intermediate = rule1 while intermediate = rule2 foreach attribute ∈ rule2 intermediate.add( attribute) evaluate( intermediate) non_dom = InsertParetoFront( intermediate ) endforeach endwhile return(P areto)

algorithm with a path-relinking procedure (path-relinking). The goal of the pathrelinking step is to explore intermediary solutions between each pair of Elite Rules in order to ensure a good Pareto Front coverage. The construction of the rules is based on the RCL. A candidate element of RCL is one pair of attribute-value. For each possible pair, the algorithm computes the TP rate and the TN rate and chooses n% of the best candidates according to the TP rate or according to the TN rate, shifted from one the other. A rule is initialized in the construction phase with a pair attribute-value randomly chosen from the RCL with uniform probability. After then, all possible candidates are evaluated and the RCL is again built. The contingency matrix is recalculated for each candidate element. The algorithm randomly selects another pair from the RCL. If this additional pair improves the TP rate or the TN rate, the pair

3 Multiobjective Optimization and Rule Learning

59

is included in the rule currently examined. Otherwise, the pair is removed from the RCL. The construction ends when the RCL is empty. Each rule is improved with the local search procedure. The algorithm randomly selects one attribute or value to replace in the current solution at each iteration. If the replacement improves the TP rate (or the TN rate), then, the new rule replaces the current one and the process continues until there is no improvement. At the end of this phase, a procedure (f ilterGeneric) checks for rules more general or speciﬁc in the Elite Rules archive before a rule is added to the archive. A rule is more speciﬁc than other rule, if it has less attribute constraints and the same contingency table. Only the more general rules are kept in the archive. The second phase applies the path-relinking between the Elite Rules and also initializes the non-dominated solutions archive (InsertParetoFront ). For each two rules, the path-relinking creates intermediate rules between them (path). Considering an initial and a terminal rule, the idea is to replace one pair attribute-value of the initial rule with a pair attribute-value of the terminal rule, at each iteration, until the former reaches the latter. Each intermediate rule is analyzed (evaluate) and if it is non-dominated by any other solution in the Elite Archive, it is included in that archive. There is no size restriction for this archive. The procedure removes solutions that are dominated by the new rule from that archive. The non-dominated solutions archive is built selecting all the non-dominated rules according to sensitivity and speciﬁcity.

3.5 Multi-objective Concepts Optimization problems that have more than one objective functions are called Multi-Objective problems. In such problems, the objectives to be optimized are usually in conﬂict with respect to each other, which means that there is no single solution for these problems. Instead, the goal is to ﬁnd good "trade-oﬀ" solutions that represent the best possible compromise among the objectives. The general multi-objective maximization problem (with no restrictions) can be stated as maximize: f (x) = (f1 (x), ..., fQ (x))

(3.9)

subjected to x ∈ X where, x is a vector of decision variables and X is a ﬁnite set of feasible solutions; fi (x), i = 1, ..., Q are the objective functions. Then, f : X → Z is a function that assigns an objective vector z = f (x) ∈ Z to each solution x ∈ X. Let 1 2 ) and z 2 = (z12 , ..., zQ ), z 1 , z 2 ∈ Z be two objective vectors. Some z 1 = (z11 , ..., zQ dominance deﬁnitions are as follows. • z 1 z 2 (z 1 dominates z 2 ) if z 1 is not worse than z 2 in any objective and is better in at least one • z 1 z 2 (z 1 strictly dominates z 2 ) if z 1 is better than z 2 in all objectives • z 1 z 2 (z 1 weakly dominates z 2 ) if z 1 is not worse than z 2 in any objective

60

C.Y. Ishida et al.

• z 1 || z 2 (z 1 and z 2 are incomparable to each other) if neither z 1 dominates z 2 nor z 2 dominates z 1 • z 1 z 2 (z 1 and z 2 are indiﬀerent) if z 1 and z 2 are equal in all objectives. The goal is to discover solutions that are not dominated by any other in the objective space. A set of non-dominated objective vectors is called Pareto optimal and the set of all non-dominated vectors is called Pareto front. The Pareto optimal set is helpful for real problem, e.g., engineering problems, and provides valuable information about the underlying problem [28]. In most applications, the search for the Pareto optimal is NP-hard [13], then the optimization problem focuses on ﬁnding a set as close as possible to the Pareto optimal. An important concept when we wish to compare algorithms is the approximation set. Let A ⊆ Z be a set of objective vectors. A is said an approximation set if any two elements of A are incomparable to each other. The dominance relations can be extended to the approximation sets. Given two approximation sets A1 and A2 , A1 is said to dominate A2 (A1 A2) if every solution vector of A2 is dominated by at least one objective vector of A1 . The other relations are deﬁned accordingly. In order to compare algorithms, it is also useful to deﬁne the relation is better than. It is said that an approximation set A1 is better than other, A2 , (A1 A2 ) if A1 A2 and A1 = A2 . A1 and A2 are said to be incomparable to each other (A1 ||A2 ) if neither A1 A2 nor A2 A1 . Those deﬁnitions are very useful when comparing the outcomes of approximation algorithms. As example, Figure 3.1 shows the Pareto front associated with the approximation set generated by the Pareto Front Elite algorithm for the data set #2(bupa). One way to evaluate the stochastic multiobjective algorithms is to compare the approximation sets samples generated by them. The Figure 3.2 shows, for data set #2(bupa), the Pareto front of PFE algorithm associated with the approximation sets of 50 independent executions of GRASP-PR Rule Learning algorithm.

Fig. 3.1. Example of approximation set of positive rules for data set #2 (bupa)

3 Multiobjective Optimization and Rule Learning

61

Fig. 3.2. Approximation sets of positive rules for data set #2 (bupa)

3.5.1

Performance Assessment

The general directions for the performance assessment methodology utilized in this work are given in the work [28]. A ﬁrst comparison is done with the dominance ranking, using the additive binary epsilon indicator [43] and the MannWhitney test [8]. Given two approximation sets, A and B, the binary epsilon indicator, Iξ (A, B) gives the minimum factor by which each point of B can be added such that the resulting set is weakly dominated by A. Given two sets of stochastic algorithms executions A = {A1 , ..., Ak } and B = {B1 , ..., Bj }, a rank is given for each approximation set Ci ∈ C, C = A ∪ B, being the number of approximation sets weakly dominating Ci plus one (equation 3.10). Thus, the smaller the rank an approximation set Ci receives the better it is. rank(Ci ) = 1 + |{Cj ∈ C : Cj Ci }|

(3.10)

Equation 3.10 provides an integer value that is assigned to each approximation set. Then, a statistical test can be performed in order to establish whether signiﬁcant diﬀerences exist between the sets A and B, in particular, whether the ranks assigned to the approximations sets that resulted of one algorithm are signiﬁcantly smaller than the ranks assigned to the approximation sets of the other algorithm. The Mann-Withney U-test, also called Mann-Whitney-Wilcoxon test or Wilcoxon rank-sum test, is a non-parametric test used to verify the null hypothesis that two samples come from the same population [8]. If the results obtained with the dominance ranking test, however, do not support conclusions about signiﬁcant diﬀerences among the investigated algorithms, then new experiments have to be done. In this work unary quality indicators were used. The unary quality indicator is a function I(X) that maps one approximation set X into a real value. Let A and B be a pair of approximation sets. The diﬀerence between their corresponding indicator values I(A) and I(B) reveals a diﬀerence in the quality of the two sets. The unary quality indicators hypervolume, unary epsilon and R2 are used in this work.

62

C.Y. Ishida et al.

The hypervolume indicator IH (A) measures the hypervolume of that portion of the objective space that is weakly dominated by an approximation set A [43]. In order to use this indicator, the objective space must be bounded. If it is not, a reference point (dominated by all points) must be deﬁned. A reference set R, in general better than the one that is being analyzed, can be utilized. If the hypervolume diﬀerence to R is considered, then smaller values correspond to higher quality of the approximation set A. The hypervolume indicator is the only unary indicator that is able to detect that one approximation set A is not better than another approximation set B for all pair of solutions [28]. The unary epsilon additive I1 (A) gives the minimum factor by which each point in R can be added such that the resulting transformed approximation set is weakly dominated by A [44]. One approximation set A is preferred than another B, according to the unary epsilon additive indicator, if I1 (A) < I1 (B). If the hypervolume and the unary epsilon additive indicate opposite preferences for two approximation sets then they are incomparable. The R2 indicator, IR2 [19] is based on a set of utility functions. A utility function is a mapping of the set Z of q-dimensional objective vectors to the set of real numbers. In this paper, the IR2 indicator used the augmented Tchebycheﬀ function given in equation 3.11, where zj∗ is a point that weakly dominates the point zj , ξ = 0.01 and λj ∈ Δ, the set of weight vectors containing (3.10) uniformly dispersed scalarizing vectors. ⎞ ⎛ n ∗ ∗ zj − zj ⎠ (3.11) uλ = − ⎝ max λj zj − zj + ξ j∈1..q

j=1

The same reference set is utilized for those indicators and, when necessary, the same reference points. The reference set is formed with the non dominated vectors of all approximation sets generated by those algorithms. The KruskalWallis statistical test [8] is utilized to compare the algorithms based on those three quality indicators. The Kruskal-Wallis test is a logical extension of the Mann-Whitney test. It is also a non-parametric test used to compare three or more samples testing the null hypothesis that all populations have identical distribution functions. For each of the two algorithms under consideration, new independent data is generated for the statistical test, with (3.10) independent runs executed for each instance. The reference points (best and worst) of each data set is determined analyzing the points in the reference set. The best and worst values of each objective is calculated. The reference point for the hypervolume indicator is formed by the worst values of each objective plus 10. The reference point z* of the R2 indicator is obtained with the best value of each objective.

3.6 Comparing Pareto Front Elite and GRASP-PR Rule Learning Algorithms Pareto Front Elite and GRASP-PR Rule Learning algorithms have the same goal: maximize the AUC criteria by the selection of the Pareto Front according

3 Multiobjective Optimization and Rule Learning

63

to sensitivity and speciﬁcity. The Pareto Front Elite with Minimum parameters set to zero can be considered an "all rule" approach, e.g., the algorithm with Minimums set to zero generates all rules for the data set. The Pareto Front of this execution is certainly the best Pareto Front according to sensitivity and speciﬁcity. The execution with no Minimum parameters, obviously, is not the best option for the large data sets, however, these Pareto fronts can be a reference for other algorithms. Then, we use all the Pareto Fronts generated by the PFE with no Minimum parameters and compare with the experiments of GRASP-PR Rule Learning algorithm. The experiments with GRASP-PR Rule Learning algorithm were done with the same methodology and paired data sets utilized in the previous section. The RCL contains 50% of the best pairs according to TP (TN) rate. Two values were tested for the number of solutions generated in the construction phase (variable maxLoop): 100 (50 positive rules and 50 negative rules) and 200 (100 positive rules and 100 negative rules). As in the Pareto Front Elite algorithm, the rules obtained with the GRASP-PR Rule Learning algorithm were used as a classiﬁer with a weighted voted classiﬁcation process based on conﬁdence. For each maxloop value, 50 independent runs of the algorithm are executed for each data set. The mean of AUC and its standard deviation for each data set are presented at Table 3.6. The second column has the values for maxLoop = 100 and the third column has the values for maxLoop = 200. The distributions of the AUC values for each data set were veriﬁed with Shapiro-Wilk test [41]. All the distributions follow a normal distribution, with exceptions of data sets #1 and #3 for GRASP-PR. Then, all the analysis were done using T-test with 95% conﬁdence level. Taking the AUC values of Pareto Front Elite algorithm with no Minimum parameters, only at data set #13, GRASP-PR algorithm has mean AUC value statistically worst than the Pareto Front Elite algorithm. At others data sets, there are no diﬀerence between the results. In others words, the meta-heuristic algorithm can obtain similar mean AUC values to all rule learning algorithm at most part of the data sets. At Table 3.6, the background colours indicate the comparison between the results of GRASP-PR Rule Learning algorithm. The values shown in the third column of table 3.6 (200 rules) are the basis for the comparison. The light gray at the background of the cells indicate that the mean AUC values for the parameter maxLoop = 200 is better than the results for maxLoop = 100. For others AUC values, there is no diﬀerence between the results. For the data sets which are not executed by PFE with no Minimum parameters, we compare the results of GRASP-PR with maxLoop = 200 with other related works. The AUC values of GRASP-PR are in third column of table 3.6 (Table 3.3). Between these data sets, GRASP-PR algorithm has similar or better AUC values in data sets #4, #9, #15 and #16. In data set #4, the AUC values of GRASP-PR is statisticaly better than C45, CN2, CN2OR, Ripper and Slipper algorithms. In data sets #9 and #16, the GRASP-PR are better than three algorithms: C45, C45NP and Ripper. In data set #16, the AUC values

64

C.Y. Ishida et al.

Table 3.6. Mean AUC for GRASP-PR Rule Learning with 100 and 200 of maximal contructed solutions

of GRASP-PR is also better than three algorithms (C45, Ripper and Slipper). The AUC values for GRASP-PR, in data set #11, are better than the results of ROCCER and C45. But, the algorithms C45NP, CN2 and CN2OR are statistically better for this data set. Just in data set #10, the GRASP-PR has the worst AUC values among the algorithms. 3.6.1

Pareto Front Analysis

The Pareto-optimal of PFE algorithm is compared with 50 executions of GRASPPR Rule Learning algorithm. Both algorithms were executed with the same methodology used in the previous experiments. The approximation sets are analysed using the dominance ranking [27] and the indicator approaches using the PISA framework [3]. At dominance ranking, each approximation set receives a rank based on the domination relation by counting the number of fronts that it dominates. The results for all data sets, including all folds, indicate that none of them, neither PFE or GRASP-PR algorithms, generates a better approximation according to dominance ranking with a signiﬁcance level of 5%. As the results with the dominance ranking were not conclusive, three unary quality indicators were used: the hypervolume [43], the epsilon [44] and the R2 [19]. Considering that the bi-objectives, sensitivity and speciﬁcity, need to be maximized, the reference point used to calculate the hypervolume and epsilon is (0, 0) and the reference point for R2 is (1, 1). The approximation sets of the Pareto Front Elite algorithm with no Minimum parameters and the GRASP-PR Rule algorithm with parameter maxLoop = 100

3 Multiobjective Optimization and Rule Learning

65

are ﬁrst tested. Then, a comparison between the same PFE version and the GRASP-PR Rule algorithm with parameter maxLoop = 200 is done. PFE All Rules X maxLoop=100 The results for positive and negative rules in Table 3.7 are analyzed with the three quality indicators. The values in Table 3.7 show the number of times, according to each indicator, that the PFE obtains signiﬁcantly better results than the GRASP-PR. Considering these indicators, the results show that the GRASPPR Rule Learning algorithm generates Pareto Fronts compared to Pareto Front Elite algorithm at ﬁve of ten data sets (#1, #2, #6, #7 and #12). However, there are four data sets (#5, #8, #13 and #14) where the meta-heuristic algorithm does not generate good fronts. Table 3.7. Number of times that PFE with no Minimum parameters has better aproximation sets of positive and negative rules than GRASP-PR Rule Learning algorithm with parameter maxLoop = 100 Positive Rules Negative Rules # hypervolume epsilon R2 hypervolume epsilon R2 1 0 0 0 0 0 0 2 0 1 1 9 10 9 3 4 4 3 0 0 0 5 10 10 7 10 10 1 6 1 2 0 9 10 0 7 0 0 0 1 0 0 8 9 10 0 10 10 8 12 0 0 0 0 0 0 13 8 9 0 0 0 0 14 10 10 5 8 9 1 Total(%) 42(0.42) 46(0.46) 16(0.16) 47(0.47) 49(0.49) 19(0.19)

The ﬁfth, sixth and seventh columns of Table 3.7 show the number of times the PFE with negative rules is preferred to the GRASP-PR Rule algorithm according to the hypervolume, epsilon and R2 indicators. The results show that the GRASP-PR algorithm presents a similar performance regarding positive and negative rules. According to the three indicators, GRASP-PR performs as well as PFE at ﬁve data sets (#1, #3, #7, #12 and #13). The last line of Table 3.7 shows the percentage of times that the PFE obtained better Pareto Fronts than the GRASP-PR in accordance with each quality indicator. The complement of the percentages presented in the last line of Table 3.7 represents the number of times the GRASP-PR obtained high quality approximation sets. Considering, independently, the hypervolume and the unary epsilon indicators, those results show that, in average, in more than 50% of its executions the heuristic algorithm presents a performance similar to the PFE,

66

C.Y. Ishida et al.

concerning the generated approximation sets. According to the R2 indicator the same conclusion is drawn for more than 80% of the GRASP-PR executions. PFE All Rules X maxLoop=200 Now, PFE algorithm is compared to GRASP-PR with the increase of the number of generated rules. For positive rules, second, third and fourth column of Table 3.8 shows the number of times that the algorithm PFE is better compared to GRASP-PR Rule algorithm with parameter maxLoop = 200 according to, respectively, hypervolume, epsilon and R2 indicators. Considering these indicators, the results show that the GRASP-PR Rule Learning algorithm generates Pareto Fronts equivalent to Pareto Front Elite algorithm at six of ten data sets. However, there are three data sets (#5, #8 and #14) where meta-heuristic algorithm does not generate good fronts in comparison with the PFE algorithm. Table 3.8. Number of times that PFE with no Minimum parameters has better Pareto Front of positive and negative rules than GRASP-PR Rule Learning algorithm with parameter maxLoop = 200 Positive Rules Negative Rules # hypervolume epsilon R2 hypervolume epsilon R2 1 0 0 0 0 0 0 2 0 0 0 8 9 8 3 3 3 3 0 0 0 5 9 3 6 10 10 1 6 1 1 0 6 9 0 7 0 0 0 1 0 0 8 6 6 0 10 10 5 12 0 0 0 0 0 0 13 0 0 0 0 0 0 14 10 10 4 1 3 1 Total(%) 29(0.29) 23(0.23) 13(0.13) 36(0.36) 41(0.41) 15(0.15)

In the ﬁfth, sixth and seventh columns, Table 3.8 shows the number of times that the approximation sets generated by the PFE are better than the ones generated by the GRASP-PR Rule algorithm with negative rules according to the hypervolume, epsilon and R2 indicators, respectively. According to these indicators, GRASP-PR gets similar results at six data sets. The last line of Table 3.8 shows the total and percentage of times that PFE has better Pareto Fronts for each indicator. The complements of these percentages show that the GRASP-PR got similar Pareto Fronts for most comparisons. Between the positive rules, more than 70% of Pareto Fronts generated by the GRASP-PR Rule Learning algorithm are similar to PFE according to hypervolume and epsilon. Between the negative rules the performance was a little below, but, still next to 65%. For the R2 indicator, 80% of the fronts are similar.

3 Multiobjective Optimization and Rule Learning

67

3.7 Conclusions The area under the ROC curve (AUC) is considered a relevant criterion to deal with imbalanced data, misclassiﬁcation costs and noisy data. Based on this preference, a question emerges: which properties must the rules have to create good classiﬁer? We work on a possible answer, a classiﬁer composed by rules having high sensitivity and speciﬁcity values. To validate this approach a previous work presented an algorithm for rule subset selection. The rule subset selection algorithm builds a Pareto Front using the sensitivity and speciﬁcity criteria selecting rules from a large set of rules. This algorithm presented good AUC values and justify the work presented here, a rule learning algorithm. The rule learning algorithm uses a Greedy Randomized Adaptive Search Procedure (GRASP) with path-relinking to create rules and to build the Pareto Front using the sensitivity and speciﬁcity criteria. The performance of the proposed algorithm was compared to the Pareto Front Elite algorithm with minimum support and minimum conﬁdence parameters set to zero for some data sets, with these parameters PFE algorithm select the pareto front between all possible rules. The experiments shows that the GRASP-PR Rule Learning generates mean AUC values similar to the PFE algorithm. And, for the execution of GRASP-PR Rule Learning with 200 generated rules, at just 30% of the positive folds and next to 41% of negative, according to the tested indicators, GRASP-PR Rule Learning algorithm does not achieve the results of Pareto Front Elite algorithm. The results between Pareto Front Elite and GRASP-PR Rule algorithms indicate no statistical diﬀerence between them. Concluding, rules with high sensitivity and speciﬁcity can be used to compose a classiﬁer which presents good AUC performance. Furthermore, these rules can be created using a multi-objective meta-heuristic approach. This approach is less expensive in processing time and memory resources than the rule subset selection algorithm, and, with the same coverage of the pareto front. However, the GRASP-PR Rule results can be improved, a depth study must be done about the inﬂuence of: the size of the RCL list, the local search strategy and the neighborhood. These aspects will be explored in future works and a parallel version of the algorithm to deal with large databases must be implemented too.

Acknowledgments We acknowledge with gratitude ﬁnancial support given by MCT-CNPq 15/2007 through Grant no. 471119/2007-5.

References 1. Agrawal, R., Srikant, R.: Fast algorithms for mining association rules. In: Bocca, J.B., Jarke, M., Zaniolo, C. (eds.) Proc. 20th Int. Conf. Very Large Data Bases, VLDB, pp. 487–499, 12–15. Morgan Kaufmann, San Francisco (1994)

68

C.Y. Ishida et al.

2. Batista, G., Milare, C., Prati, R.C., Monard, M.: A comparison of methods for rule subset selection applied to associative classiﬁcation. Inteligencia Artiﬁcial. Revista Iberoamericana de IA (32), 29–35 (2006) 3. Bleuler, S., Laumanns, M., Thiele, L., Zitzler, E.: PISA – A platform and programming language independent interface for search algorithms. In: Fonseca, C.M., Fleming, P.J., Zitzler, E., Deb, K., Thiele, L. (eds.) EMO 2003, vol. 2632, pp. 494– 508. Springer, Heidelberg (2003) 4. Clark, P., Niblett, T.: The CN2 induction algorithm. Machine Learning 3, 261–283 (1989) 5. Clark, P., Niblett, T.: Rule induction with cn2: Some recent improvements. In: ECML: European Conference on Machine Learning. Springer, Berlim (1991) 6. Cohen, W.W.: Fast eﬀective rule induction. In: ICML, pp. 115–123 (1995) 7. Cohen, W.W., Singer, Y.: A simple, fast, and eﬀective rule learner. In: Proceedings of the 6th National Conference on Artiﬁcial Intelligence (AAAI 1999); Proceedings of the 11th Conference on Innovative Applications of Artiﬁcial Intelligence, July 18–22, pp. 335–342. AAAI/MIT Press (1999) 8. Conover, W.J.: Practical nonparametric statistics. Wiley, Chichester (1971) 9. de la Iglesia, B., Philpott, M.S., Bagnall, A.J., Rayward-Smith, V.J.: Data mining rules using multi-objective evolutionary algorithms. In: Congress on Evolutionary Computation, pp. 1552–1559. IEEE Computer Society, Los Alamitos (2003) 10. de la Iglesia, B., Reynolds, A., Rayward-Smith, V.J.: Developments on a multiobjective metaheuristic (MOMH) algorithm for ﬁnding interesting sets of classiﬁcation rules. In: Coello Coello, C.A., Hernández Aguirre, A., Zitzler, E. (eds.) EMO 2005. LNCS, vol. 3410, pp. 826–840. Springer, Heidelberg (2005) 11. Newman, C.B.D.J., Merz, C.: UCI repository of machine learning databases (1998) 12. Domingos, P.: Unifying instance-based and rule-based induction. Machine Learning 24(2), 141–168 (1996) 13. Ehrgott, M.: Approximation algorithms for combinatorial multicriteria optimization problems. International Transactions in Operational Research 7, 5–31 (2000) 14. Fawcett, T.: Using rule sets to maximize ROC performance. In: Cercone, N., Lin, T.Y., Wu, X. (eds.) ICDM, pp. 131–138. IEEE Computer Society, Los Alamitos (2001) 15. Feo, T.A., Resende, M.G.C.: Greedy randomized adaptive search procedures. Journal of Global Optimization 6, 109–133 (1995) 16. Ferri, C., Flach, P., Hernandez-Orallo, J.: Learning decision trees using the area under the ROC curve. pp. 139–146 (July 2002) 17. Festa, P., Resende, M.G.C.: GRASP: An annotated bibliography. Technical report, AT& T Labs Research, Florham Park, NJ (January 2001) 18. Glover, F.: Tabu search and adaptive memory programming - advances, applications and challenges. In: Barr, R.S., Helgason, R.V., Kennington, J.L. (eds.) Interfaces in Computer Science and Operations Research, pp. 1–75. kluwer, Dordrecht (1996) 19. Hansen, M.P., Jaszkiewicz, A.: Evaluating the quality of approximations to the non-dominated set. Technical Report IMM-REP-1998-7, Technical University of Denmark (March 1998) 20. Ishibuchi, H.: Multiobjective association rule mining. In: PPSN Workshop on Multiobjective Problem Solving from Nature, pp. 39–48. Reykjavik, Iceland (2006) 21. Ishibuchi, H., Nojima, Y.: Accuracy-complexity tradeoﬀ analysis by multiobjective rule selection. In: ICDM, pp. 39–48. IEEE Computer Society, Los Alamitos (2005)

3 Multiobjective Optimization and Rule Learning

69

22. Ishida, C.Y., de Carvalho, A.B., Pozo, A.T.R., Goldbarg, E.F.G., Goldbarg, M.C.: Exploring multi-objective pso and grasp-pr for rule induction. In: Eighth European Conference on Evolutionary Computation in Combinatorial Optimisation. Springer, Heidelberg (to appear, 2008) 23. Ishida, C.Y., Pozo, A.T.R.: Optimization of the auc criterion for rule subset selection. In: Intelligent Systems Design and Applications, 7th. International Conference on Intelligent Systems Design and Application. IEEE Computer Society, New York (2007) 24. Ishida, C.Y., Pozo, A.T.R.: Pareto front elite. In: XXVII Congresso SBC 2007. ENIA VI Encontro Nacional de Inteligencia Artiﬁcial (2007) 25. Jin, Y.: Multi-Objective Machine Learning. Springer, Berlin (2006) 26. Jovanoski, V., Lavrač, N.: Classiﬁcation rule learning with APRIORI-C. In: Brazdil, P.B., Jorge, A.M. (eds.) EPIA 2001. LNCS (LNAI), vol. 2258, pp. 44– 51. Springer, Heidelberg (2001) 27. Knowles, J., Thiele, L., Zitzler, E.: A tutorial on the performance assessment of stochastic multiobjective optimizers. 214, Computer Engineering and Networks Laboratory (TIK), Swiss Federal Institute of Technology (ETH) Zurich (July 2005) 28. Knowles, J., Thiele, L., Zitzler, E.: A Tutorial on the Performance Assessment of Stochastic Multiobjective Optimizers. 214, Computer Engineering and Networks Laboratory (TIK), ETH Zurich, Switzerland, revised version (Feburary 2006) 29. Laguna, M., Marti, R.: Grasp and path relinking for 2-layer straight line crossing minimization. INFORMS J. on Computing 11(1), 44–52 (1999) 30. Lavrač, N., Flach, P., Zupan, B.: Rule evaluation measures: A unifying view. In: Džeroski, S., Flach, P.A. (eds.) ILP 1999. LNCS (LNAI), vol. 1634, pp. 174–185. Springer, Heidelberg (1999) 31. Li, W., Han, J., Pei, J.: CMAR: Accurate and eﬃcient classiﬁcation based on multiple class-association rules. In: Cercone, N., Lin, T.Y., Wu, X. (eds.) ICDM, pp. 369–376. IEEE Computer Society, Los Alamitos (2001) 32. Liu, B., Hsu, W., Ma, Y.: Integrating classiﬁcation and association rule mining. In: Knowledge Discovery and Data Mining, pp. 80–86 (1998) 33. Prati, R.C., Flach, P.A.: ROCCER: An algorithm for rule learning based on ROC analysis. In: Kaelbling, L.P., Saﬃotti, A. (eds.) IJCAI, pp. 823–828. Professional Book Center (2005) 34. Provost, F., Domingos, P.: Tree induction for probability based ranking. Machine Learning 52(3), 199–215 (2003) 35. Provost, F.J., Fawcett, T.: Analysis and visualization of classiﬁer performance: Comparison under imprecise class and cost distributions. In: KDD, pp. 43–48 (1997) 36. Quinlan, J.: C4. 5: Programs for Machine Learning. Morgan Kaufmann, San Francisco (1993) 37. Quinlan, J.R.: C4.5: Programs for Machine Learning. Morgan Kaufmann, San Francisco (1992) 38. Rakotomamonjy, A.: Optimizing area under roc curve with SVMs. In: HernándezOrallo, J., Ferri, C., Lachiche, N., Flach, P.A. (eds.) ROCAI, pp. 71–80 (2004) 39. Resende, M., Ribeiro, C.: Greedy randomized adaptive search procedures. In: Glover, F., Kochenberger, G. (eds.) Handbook of Metaheuristics, pp. 219–249. Kluwer Academic Publishers, Dordrecht (2002) 40. Sebag, A., Lucas: ROC-based evolutionary learning: Application to medical data mining. In: International Conference on Artiﬁcial Evolution, Evolution Artiﬁcielle. LNCS, vol. 6 (2003)

70

C.Y. Ishida et al.

41. Shapiro, S.S., Wilk, M.B.: An analysis of variance test for normality (complete samples). Biometrika 52(3-4), 591–611 (1965) 42. Yin, X., Han, J.: Cpar: Classiﬁcation based on predictive association rules. In: Proceedings SIM International Conference on Data Mining (SDM 2003), pp. 331– 335 (2003) 43. Zitzler, E., Thiele, L.: Multiobjective Evolutionary Algorithms: A Comparative Case Study and the Strength Pareto Approach. IEEE Transactions on Evolutionary Computation 3(4), 257–271 (1999) 44. Zitzler, E., Thiele, L., Laumanns, M., Fonseca, C.M., da Fonseca, V.G.: Performance assessment of multiobjective optimizers: an analysis and review. IEEE-EC 7, 117–132 (2003)

4 Clustering Dynamic Web Usage Data Alzennyr da Silva1 , Yves Lechevallier1 , Fabrice Rossi1 , and Francisco de Carvalho2 1

2

Project AxIS, INRIA-Rocquencourt Domaine de Voluceau, B.P. 105 78153 Le Chesnay cedex, France {Alzennyr.Da_Silva,Yves.Lechevallier,Fabrice.Rossi}@inria.fr Centro de Informatica - CIn / UFPE Av. Prof. Luiz Freire, s/n, CDU 50740-540 Recife, Brazil [email protected]

Most classiﬁcation methods are based on the assumption that data conforms to a stationary distribution. The machine learning domain currently suﬀers from a lack of classiﬁcation techniques that are able to detect the occurrence of a change in the underlying data distribution. Ignoring possible changes in the underlying concept, also known as concept drift, may degrade the performance of the classiﬁcation model. Often these changes make the model inconsistent and regular updatings become necessary. Taking the temporal dimension into account during the analysis of Web usage data is a necessity, since the way a site is visited may indeed evolve due to modiﬁcations in the structure and content of the site, or even due to changes in the behavior of certain user groups. One solution to this problem, proposed in this article, is to update models using summaries obtained by means of an evolutionary approach based on an intelligent clustering approach. We carry out various clustering strategies that are applied on time sub-periods. To validate our approach we apply two external evaluation criteria which compare diﬀerent partitions from the same data set. Our experiments show that the proposed approach is eﬃcient to detect the occurrence of changes.

4.1 Introduction Web Mining [7] appeared at the end of 90s and consists in using Data Mining techniques in order to develop methods that allow relevant information to be extracted from Web data (such as documents, interaction traces, link structure, etc.). A more specialized branch of this domain, called Web Usage Mining (WUM) [2] [14], deals with techniques based on Data Mining that are applied to the analysis of users behavior in a website. The present article is placed is this last context. N. Nedjah et al. (Eds.): Innovative Applications in Data Mining, SCI 169, pp. 71–82. c Springer-Verlag Berlin Heidelberg 2009 springerlink.com

72

A. da Silva et al.

WUM consists in extracting interesting information from ﬁles which register Web usage traces. This activity has become very critical for eﬀective website management. In the e-commerce domain, for example, one of the most important motivations for the analysis of usage is the need to build up consumer loyalty and to make the site more appealing to new visitors. Accurate Web usage information could help to attract new customers, retain current customers, improve cross marketing/sales, measure the eﬀectiveness of promotional campaigns, track leaving customers and ﬁnd the most eﬀective logical structure for their Web space. Other applications we can cite are the creation of adaptive websites, support services, personalization, network traﬃc ﬂow analysis, etc. Most traditional methods in this domain take into account the entire period during which usage traces were recorded, the results obtained naturally being those which prevail over the total period. Consequently, certain types of behaviors, which take place during short sub-periods are not detected and thus remain undiscovered by traditional methods. It is, however, important to study these behaviors and thus carry out an analysis related to signiﬁcant time sub-periods. It will then be possible to study the temporal evolution of users’ proﬁles by providing descriptions that can integrate the temporal aspect. The access patterns to Web pages are indeed of a dynamic nature, due both to the on-going changes in the content and structure of the website and to changes in the users’ interest. The access patterns can be inﬂuenced by certain parameters of a temporal nature, such as the time of the day, the day of the week, recurrent factors (summer/winter vacations, national holidays, Christmas) and non-recurrent global events (epidemics, wars, economic crises, the World Cup). Furthermore, as the volume of mined data is great, it is important to deﬁne summaries to represent user proﬁles. WUM has just recently started to take account of temporal dependence in usage patterns. In [11], the authors survey the work to date and explore the issues involved and the outstanding problems in temporal data mining by means of a discussion about temporal rules and their semantic. In addition, they investigate the conﬂuence of data mining and temporal semantics. Recently in [8], the authors outline methods for discovering sequential patterns, frequent episodes and partial periodic patterns in temporal data mining. They also discuss techniques for the statistical analysis of such approaches. Notwithstanding these considerations, the majority of methods in WUM are applied over the entire period that covers all the available data. Consequently, these methods reveal the most predominant behaviors in data, and the interesting short-term behaviors which may occur during short periods of time are not taken into account. For example, when the data analysed is inserted into a dynamic domain related to a potential long period of time (such as in the case of Web log ﬁles), it is to be expected that behaviors evolve over time. These considerations have given rise to many studies in data analysis, especially concerning the adaptation of traditional static data-based methods to the dynamic data framework. In this line of research, our proposition is to use summaries obtained by an evolutionary clustering approach applied over time

4 Clustering Dynamic Web Usage Data

73

sub-periods to carry out a follow-up of the user proﬁle evolution. The main objective of this article is to propose and evaluate a monitoring clustering strategy which is able to ﬁnd changes in a stream of Web usage data. This chapter is organized as follows. Section 4.2 describes the proposed clustering approach based on time sub-periods. Section 4.3 presents the experimental framework including the benchmark data set analysed, the algorithm and the external evaluation criteria adopted as well as a discussion on the results obtained. The last section reports the ﬁnal conclusion and some suggestions for future work.

4.2 Clustering Approach Based on Time Sub-periods In unsupervised classiﬁcation modelling of dynamic data, new clusters may emerge at any time and existing clusters may evolve or disappear. The clustering problem can be seen as an evolutionary process of detecting and tracking dynamic clusters. The approach proposed in this article consists initially in splitting the entire time period analysed into more signiﬁcant sub-periods. In our experiments, we use the months of the year to deﬁne the time sub-periods (cf. section 4.3.4). This is done with the aim of discovering the evolution of old patterns or the emergence of new ones. This fact would not have been revealed by a global analysis over the whole time period. After this ﬁrst step, a clustering method is applied on the data of each time sub-period, as well as over the complete period. The results provided for each clustering are then compared. The main base of our proposition regards the speciﬁcation of a compact structure to summarize the clustering of data within a time sub-period without storing the entire data set. This structure is designated by the cluster centre (prototype) and represents a user proﬁle. These prototypes reﬂect the behaviour of individuals belonging to the same cluster. The following subsections describe in detail the four clustering strategies we study. 4.2.1

Global Clustering

This clustering strategy corresponds to the traditional practice, that is to say, the clustering algorithm is applied on the entire data set containing all the individuals, without taking into account the temporal information. After applying the clustering algorithm, we have a partition containing K clusters. We then apply a ﬁlter on each of these clusters in order to deﬁne new sub-clusters containing individuals belonging to the same time sub-periods (for example, the same month of the year) (cf. Figure 4.1). After that we put together sub-clusters belonging to the same time sub-periods, which deﬁnes one partition per sub-period. We will then compare these last partitions with those obtained by the other clustering strategies.

74

A. da Silva et al.

Fig. 4.1. Global clustering

Fig. 4.2. Independent local clustering

4 Clustering Dynamic Web Usage Data

4.2.2

75

Independent Local Clustering

In this clustering strategy, we ﬁrst split the data set by time sub-period. After that, we apply the clustering algorithm on each sub-period individually, which gives us as many partitions as there are time sub-periods. At the end of this process we will have a partition containing K clusters in each time sub-period, each independent from the other. (cf. Figure 4.2). 4.2.3

Previous Local Clustering

In this clustering strategy, we begin by applying the clustering algorithm in the ﬁrst time sub-period, which gives us a partition into K clusters. We take the prototype of each of these clusters in order to deﬁne a new partition on the individuals belonging to the next time sub-period. To do so, we apply the clustering allocation phase. We repeat this process for all the following time sub-periods (cf. Figure 4.3). 4.2.4

Dependent Local Clustering

This clustering strategy has some points in common with the previous one. The main diﬀerence here is that we use the prototypes obtained by the clustering on the previous time sub-periods to initiate a completely new clustering process rather than applying only the allocation phase. In other words, we run the algorithm until its convergence.

Fig. 4.3. Previous local clustering

76

A. da Silva et al.

Fig. 4.4. Dependent local clustering

4.3 Experimental Framework 4.3.1

Usage Data

As a case study, we use a benchmark website from Brazil1 . This site contains a set of static pages (details of teaching staﬀ, academic courses, etc.) and dynamic pages (see [4] [3] [12] [13] for an analysis of this part of the site). We studied the accesses to the site from 1st July 2002 to 31st May 2003. For Web usage data pre-processing, we adopt the methodology proposed by [15] who deﬁnes a navigation as a succession of requests not more than 30 minutes apart, coming from the same user. In order to analyse the more representative traces of usage, we selected long navigations (containing at least 10 requests and with a total duration of at least 60 seconds) which are assumed to have originated from human users (the ratio between the duration and number of requests must be at least 4, which means a maximum of 15 requests per minute). This was done in order to extract human navigations and exclude those which may well have come from Web robots. The elimination of short navigations is justiﬁed by the search for usage patterns in the site rather than simple accesses which do not generate a trajectory in the site. After ﬁltering and eliminating outliers, we obtained a total of 138,536 navigations. 4.3.2

Clustering Algorithm

Our method uses an adapted version of the dynamic clustering algorithm [1] [5] [9] applied on a data table containing the navigations in its rows and real-value variables in its columns (cf. table 4.1). As a distance measure, we adopt the 1

This website is available at the following address: http://www.cin.ufpe.br/

4 Clustering Dynamic Web Usage Data

77

Table 4.1. Description of the variables describing navigations No 1 2 3 4 5 6 7 8 9 10 11 12 13 14

Field IDNavigation NbRequests_OK NbRequests_BAD PRequests_OK NbRepetitions PRepetitions TotalDuration AvDuration AvDuration_OK

Description Navigation code Number of successful requests (status = 200) in the navigation Number of failed requests (status = 200) in the navigation Percentage of successful requests ( = NbRequests_OK/ NbRequests) Number of repeated requests in the navigation Percentage of repetitions ( = NbRepetitions / NbRequests) Total duration of the navigation (in seconds) Average of duration ( = TotalDuration / NbRequests) Average of duration among successful requests ( = TotalDuration_OK/NbRequests_OK) NbRequests_SEM Number of requests related to pages in the site’s semantic structure PRequests_SEM Percentage of requests related to pages in the site’s semantic structure (=NbRequests_Sem/ NbRequests) TotalSize Total size of transferred bytes in the navigation AvTotalSize Average of transferred bytes ( = TotalSize / NbRequests_OK) MaxDuration_OK Duration of the longest request in the navigation (in seconds)

Euclidean distance. For all the experiments, we deﬁned an a priori number of clusters equal to 10 with a maximum number of iterations equal to 100. The number of random initialisations is equal to 100, except when the algorithm is initialised with the results obtained from a previous execution (for the strategies presented in section 4.2.3 and section 4.2.4). 4.3.3

Evaluation Criteria

To analyse the results, we apply two external criteria. This implies that we evaluate the results of a clustering algorithm based on how well the clustering matches prespeciﬁed gold standard classes. For a cluster-by-cluster analysis, we compute the F-measure [16]. To compare two partitions, we look for the best representation of the cluster a in the ﬁrst partition by a cluster b in the second partition, i.e., we look for the best match between the clusters of two given partitions. This gives us as many values as there are clusters in the ﬁrst partition. The F-measure combines in a single value the measures of precision P and recall R between an a priori partition containing C clusters and the partition reached by the clustering method containing Q clusters . Let V = {v1 , . . . , vc , . . . , vC } be the a priori partition and U = {u1 , . . . , uq , . . . , uQ } the partition reached by the clustering method. Their contingency table is depicted in Figure 4.5. The F-measure is then deﬁned as follows: F =

C ni i=1

F (i, j) =

n

maxj=1,...,Q F (i, j), where

2P (i, j)R(i, j) P (i, j) + R(i, j)

(4.1)

78

A. da Silva et al.

Fig. 4.5. Contingency table

In this formula, ni indicates the number of elements in the cluster vi and n is the total number of elements. For a global analysis, we apply the corrected Rand (CR) index [6] to compare two partitions. The CR index also assesses the degree of agreement between an a priori partition and a partition furnished by the clustering algorithm. We use the CR index because it is not sensitive to the number of clusters in the partitions or to the distributions of the individuals in the clusters. The CR is deﬁned as: Q C nij

CR =

i=1 j=1 Q 1 ni. 2[ 2 i=1

2

−

Q C n−1 ni. n.j 2

i=1

2

j=1

2

Q C C n−1 n.j ni. n.j + 2 ]− 2 2 2 j=1

i=1

(4.2)

j=1

, nij represents the number of objects that are in clusters where n2 = n(n−1) 2 ui and vj , ni. indicates the number of objects in cluster ui , n.j indicates the number of objects in cluster vj and n is the total number of objects in the data set. The F-measure takes a value in the range [0,+1], whereas the corrected Rand index values are in the range [-1,+1]. In both cases, the value 1 indicates a perfect agreement and values near 0 correspond to cluster agreements found by chance. In fact, an analysis made by [10] conﬁrmed corrected Rand index values near 0 when presented to clusters generated from random data, and showed that values lower than 0.05 indicate clusters achieved by chance. 4.3.4

Results

Figure 4.6 depicts the CR values obtained from the confrontation of the resulting partitions reached by the four diﬀerent clustering strategies presented in section 4.2. For the F-measure, Figure 4.7 presents boxplots summarizing the 10 values obtained (one value per cluster) on each month analysed. The partitions compared are marked in red in Figure 4.1, Figure 4.2, Figure 4.3 and Figure 4.4.

4 Clustering Dynamic Web Usage Data

79

In Figure 4.6, the values of the CR index reveal that the results from the local independent clustering are very diﬀerent from those of the local dependent and global clustering (low values of CR means that the partitions compared are divergent). In other words, we can see by the confrontation of the local independent clustering versus the local dependent and global clustering that there are almost always low values, i.e., the local independent clustering is able to ﬁnd certain clusters that were not detected by the other two clustering strategies. These conclusions are also conﬁrmed by the F-measure (cf. Figure 4.7). We can also notice that the local previous clustering does not give very different results from those obtained by the local dependent clustering. The same conclusion is also valid for the confrontation between the local dependent clustering and global clustering. Using a cluster-by-cluster confrontation via the F-measure, we reﬁne the analysis (see Figure 4.7). What appears quite clearly is that the clusters are very stable over time if we apply either the local previous, local dependent or global clustering strategies. In fact, no value is lower than 0.877, which represents a very good score. On the other hand, in the case of local independent clustering,

Fig. 4.6. Corrected Rand index values computed partition by partition

80

A. da Silva et al.

Fig. 4.7. F-measure values computed cluster by cluster

we detect clusters that are very diﬀerent from those obtained by the global and local dependent clustering (some values are lower than 0.5). What is surprising is that partitions obtained by the local dependent clustering are very similar to those obtained from the global clustering. We could thus speculate whether an analysis carried out on time sub-periods would be able to obtain results supposed to be revealed by a global analysis on the entire data set. To summarize, we can say that the local dependent clustering method shows that the clusters obtained change very little or do not change at all, whereas the local independent clustering method is more sensitive to changes which occur from one time sub-period to another.

4.4 Conclusion In the present work, we addressed the problem of processing dynamic data in the WUM domain. The issues discussed highlight the need to deﬁne or adapt methods to extract knowledge and to follow the evolution of this kind of data. Although many powerful knowledge discovery methods have been proposed for

4 Clustering Dynamic Web Usage Data

81

WUM, very little work has been devoted to handling problems related to data that can evolve over time. In this article, we have proposed a divide and conquer based approach in the sense that we split the original data set into temporal windows regarding the position of the data in the time scale. Our experiments have shown that the analysis of dynamic data by independent time sub-periods oﬀers a certain number of advantages such as making the method sensitive to cluster changes over time. Analysing changes in clusters of usage data over time can provide important clues about the changing nature of how a website is used, as well as the changing loyalties of its users. Furthermore, as our approach splits the data and concentrates the analysis on fewer sub-sets, some constraints regarding hardware limitations could be overcome. Possible future work could involve the application of other clustering algorithms and the implementation of techniques that enable the automatic discovery of the number of clusters as well as identifying cluster fusions and splits over time.

Acknowledgements The authors are grateful to CAPES (Brazil) and the collaboration project between INRIA (France) and FACEPE (Brazil) for their support for this research.

References 1. Anderberg, M.R.: Cluster analysis for applications. In: Probability and Mathematical Statistics. Academic Press, New York (1973) 2. Cooley, R., Mobasher, B., Srivastava, J.: Data preparation for mining world wide web browsing patterns. Journal of Knowledge and Information Systems 1(1), 5–32 (1999) 3. Da Silva, A., De Carvalho, F., Lechevallier, Y., Trousse, B.: Characterizing visitor groups from web data streams. In: Proceedings of the 2nd IEEE International Conference on Granular Computing (GrC 2006), pp. 389–392, May 10- 12 (2006) 4. Da Silva, A., De Carvalho, F., Lechevallier, Y., Trousse, B.: Mining web usage data for discovering navigation clusters. In: 11th IEEE Symposium on Computers and Communications (ISCC 2006), pp. 910–915 (2006) 5. Diday, E., Simon, J.C.: Clustering analysis. In: Fu, K. (ed.) Digital Pattern Classification, pp. 47–94. Springer, Heidelberg (1976) 6. Hubert, L., Arabie, P.: Comparing partitions. Journal of Classification 2, 193–218 (1985) 7. Kosala, R., Blockeel, H.: Web mining research: A survey. ACM SIGKDD Explorations: Newsletter of the Special Interest Group on Knowledge Discovery and Data Mining 2, 1–15 (2000) 8. Laxman, S., Sastry, P.S.: A survey of temporal data mining. SADHANA - Academy Proceedings in Engineering Sciences, Indian Academy of Sciences 31(2), 173–198 (2006)

82

A. da Silva et al.

9. MacQueen, J.: Some methods for classification and analysis of multivariate observations. In: 5th Berkley Symposium on Mathematics and Probability, vol. 1, pp. 281–297 (1967) 10. Milligan, G.W., Cooper, M.C.: A study of the comparability of external criteria for hierarchical cluster analysis. Multivariate Behavioral Research 21(4), 441–458 (1986) 11. Roddick, J.F., Spiliopoulou, M.: A survey of temporal knowledge discovery paradigms and methods. IEEE Transactions on Knowledge and Data Engineering 14(4), 750–767 (2002) 12. Rossi, F., De Carvalho, F., Lechevallier, Y., Da Silva, A.: Comparaison de dissimilarités pour l’analyse de l’usage d’un site web. In: Actes des 6me journes Extraction et Gestion des Connaissances (EGC 2006), Revue des Nouvelles Technologies de l’Information (RNTI-E-6), vol. II, pp. 409–414 (January 2006) 13. Rossi, F., De Carvalho, F., Lechevallier, Y., Da Silva, A.: Dissimilarities for web usage mining. In: Actes des 10me Confrence de la Fdration Internationale des Socits de Classification (IFCS2006) (July 2006) 14. Spiliopoulou, M.: Data mining for the web. In: Workshop on Machine Learning in User Modelling of the ACAI 1999, pp. 588–589 (1999) 15. Tanasa, D., Trousse, B.: Advanced data preprocessing for intersites web usage mining. IEEE Intelligent Systems 19(2), 59–65 (2004) 16. van Rijsbergen, C.J.: Information Retrieval, 2nd edn. Butterworths, London (1979)

5 Towards Characterization of the Data Generation Process Vasudha Bhatnagar and Sarabjeet Kochhar Department of Computer Science, University of Delhi, Delhi-07, India {vbhatnagar,skochhar}@cs.du.ac.in

Data Mining applications have found interesting applications in commercial and scientiﬁc domains. Last two decades have seen rapid strides in development of elegant algorithms that induce useful predictive and descriptive models from large data repositories available widely. In last decade serious eﬀort has also been made towards mining of evolving data-sets and now several one pass algorithms with restricted memory footprints are available for use in data stream environments. Study of temporal evolution of the patterns has been recognized as an important next generation data mining problem by both - the research and user communities. Comparative analyses of the changes detected in the discovered trends over the temporal dimension are likely to provide an insight into the dynamics of the dgp. Diﬀerent levels of abstractions from the end-user’s viewpoint, form the second dimension of such analyses. In this chapter, we study the problem of characterizing the data generation process underlying an evolving dataset (i.e. data stream). We propose a generic architecture for this purpose, consisting of an online and an oﬄine component. The online component mines the data stream and employs consolidation to derive metrics which qualify changes in the discovered patterns. The oﬄine component is invoked on demand to draw inferences about pre-speciﬁed characteristics of dgp using a set of operators. We instantiate the architecture with a system which characterizes the dgp for consistency w.r.t. the support behavior of streaming itemsets and also present its two applications.

5.1 Introduction Though database technology is ubiquitous in today’s world, understanding of the data generation process (dgp) in most real life databases is abysmally low. It is important to understand the nature of the data generating process and it’s inﬂuencing forces in order to develop capabilities to control them [11]. The N. Nedjah et al. (Eds.): Innovative Applications in Data Mining, SCI 169, pp. 83–105. c Springer-Verlag Berlin Heidelberg 2009 springerlink.com

84

V. Bhatnagar and S. Kochhar

best that we have in the experiences, knowledge and intuition of the experts in the respective areas is known as the domain knowledge. However, the models of domain knowledge are too inadequate to predict the course of dgp or to describe its nature. Study and analysis of changes in the evolving patterns is an important way to understand the dgp [12, 13, 30]. The following scenarios present a small sample of diﬀerent environments in which one necessarily needs to look for changes in trends in the discovered patterns. • Since the tsunami tragedy of Dec. 2004, the seismologists have wanted to analyze the seismic data collected from all over the globe to study i) how the patterns in data changed on hourly, daily, weekly, monthly basis prior to the occurrence of the earthquake, ii) how did the changes diﬀer from those discovered from data collected at diﬀerent locations, iii) were similar changes observed ever before in the recorded historical data? • A FMCG company wants to compare the sale trends of selected goods in different population groups in diﬀerent geographical locations over time to ﬁnd out i) which age groups have shown signiﬁcant changes in buying patterns, ii) how the brand loyalties in diﬀerent age groups have changed over a time period? • A hospital admission database is to be analyzed to discover i) the changing prevalence of diseases over time, across diﬀerent cross-sections of populations, ii) changes in the recovery trends as a result of introduction of new drugs in certain pathological conditions, iii) changes in the socio-economic patterns of the victims of a disease. The dgp is a function of multiple parameters - external and/or internal to the system and their complex interplay. Often some of these parameters may be unknown, making complete understanding of dgp an awesome task. Moreover, the dgp can be characterized on the basis of diﬀerent measures of interestingness viz. novelty, stability, action-ability, unexpectedness etc. [21, 32, 33, 35]. All these measures are highly subjective with respect to both user and time, which makes the task of dgp characterization even more challenging. Instead of characterizing the dgp completely, we orient our objective towards a modest proposal to analyze a small subset of dgp characteristics. The databases commonly found in applications such as networks, telecommunication, web log, sensor data etc. keep growing and accumulating, and are usually maintained as archives, hiding nuggets of knowledge in their bowels [5, 15, 29]. Such evolving databases present opportunities to study the changes in patterns discovered at periodic intervals and are likely to reveal some characteristics inﬂuencing the dgp [9, 11]. These characteristics provide an insight into how’s, why’s and when’s of the process. 5.1.1

The Problem

Let Di be the data repository at time instance ti , i = 0, 1, 2, . . ., where granularity of time is user deﬁned. Unpopulated database at time instance t0 is denoted

5 Towards Characterization of the Data Generation Process

85

by D0 . The evolution of the database from D0 to Dk (at time instance tk ) can Δ

Δ

Δ

Δ

1 2 3 k be represented by transitions D0 −→ D1 −→ D2 −→ . . . −→ Dk . . ., where Δi = Di − Di−1 , (i > 1) is the incremental database evolved during the time interval ([ti−1 , ti [). Let K0 , K1 , . . . , Ki be the trends mined from databases D0 , D1 , . . . , Di respectively. We study the problem of drawing inferences about the nature of underlying data generation process by detecting changes in the Ki s over a speciﬁed period of time. The following issues need to be addressed:

i) Specify what feature(s) of the dgp are to be characterized (volatility, simplicity, novelty, actionability etc.). Some of the features may be interdependent and hence diﬃcult to characterize. ii) Determine what type of changes will reveal the target feature(s). iii) Model the determined changes in form of metrics that can be captured from the discovered patterns. iv) Quantify changes from the stored metrics and infer the target characteristic(s) of the dgp. 5.1.2

The Approach

A generic solution to the above mentioned problem is outlined in Figure 5.1 as a continuous monitoring system with an online and oﬄine component. The online component continuously mines the streaming data and produces knowledge units (Ki s). It also captures changes between the Ki s, derives certain predeﬁned metrics by consolidation mechanism and stores these metrics as snapshots. The oﬄine component is invoked either on demand by the user (in the manual mode) or periodically (in the automated mode) to ﬁnd time variant characteristics of the dgp.

D mine K 0 0 mine D1 K1

Derive Change Metrics

Snapshots

Deduce Target Characterstics

D mine K i i The online component

The offline component

Fig. 5.1. Tasks in the process of characterizing data generation process

The following observations make generality of the proposed approach evident: 1. For the desired type of descriptive or predictive model (Association Rules, Classiﬁcation model or Clustering schemes), an appropriate stream mining algorithm can be selected to discover Ki s

86

V. Bhatnagar and S. Kochhar

2. Based on the type of changes to be studied, change metrics can be designed to be captured by the online component. 3. Depending on the desired model and type of changes, mechanisms can be designed to aid semantic interpretation of the changes recorded by the online component. This aids in deducing the target characteristics. Organization: Section 5.2 presents related works. A generic architecture of a system for inferencing dgp characteristics is presented in Section 5.3. SI-CHAR system, an instantiation of the generic architecture is presented in next two sections. Section 5.4 presents the online component of SI-CHAR system; Section 5.5 presents its oﬄine component. Two applications are presented in Sections 5.6 and 5.7. Section 5.8 concludes the paper, giving direction for future works.

5.2 Related Works The idea of continuous monitoring of evolving databases is inspired by [7], which proposes I-MIN model for carrying out KDD process. Design of tailored applications [23] for continuous monitoring in I-MIN model has inﬂuenced the proposal presented in this chapter. A work philosophically same as the work presented in this chapter can be found in [34]. The paper proposes a cluster transition model has been to track changes in clusters. The model can accommodate any clustering algorithm to detect and monitor changes, with the objective of gaining insights into the nature of underlying data stream. To the best of author’s knowledge, characterization of dgp has not been attempted earlier. However a large body of work in the area of change detection and concept drifts provide feasibility for our attempt, and are noted below. In the machine learning community, "concept drifts" have been studied extensively [4]. The problem of concept drifts deals with the inﬂuence of changing data on the learning process and attempt is made to design algorithms that adapt to the learning process with the changing data characteristics. Focus framework [25], is a signiﬁcant beginning for quantifying the diﬀerence between two data-sets in terms of the models they induce. The DEMON framework [24] addresses data evolution by detecting changes in data along temporal dimension in order to update the knowledge base. However it is only with the increasing interest in data streams, that the problem of detecting and modeling changes in data streams has become hot [1, 3, 6, 16, 18, 19, 28, 36, 37]. An algorithm based on creation of temporal and spatial velocity proﬁles has been proposed for detecting and visualizing changes in spatio-temporal data streams in [1]. Methods for detecting changes in data streams with a proven statistical signiﬁcance have been proposed in [6]. A decision tree based approach for on-demand discovery of distribution and concept drifts in data streams has been discussed in [19]. An algorithm based on correspondence tracing studies changes in two classiﬁers possibly generated at two diﬀerent time instances has been proposed in [37]. A quick, limited memory requirement algorithm to ﬁnd changes in network data streams can be found in [14]. An algorithm to detect

5 Towards Characterization of the Data Generation Process

87

concept drifts has been presented in [18]. However, the emphasis in the cited works is on change detection and not on understanding of detected changes.

5.3 The Generic Architecture The DGP-CHAR system shown in Figure 5.2 is designed to accomplish the tasks shown in Figure 5.1. The system consists of an online component, oﬄine component and a focusing unit. The online component is composed of the mining unit and consolidation unit while the oﬄine component consists of a characterization unit. The focusing unit allows user to set focus of monitoring and change detection and communicates user interest to the mining unit and THE characterization unit. DGP

User Input

Evolving Database

Focusing Unit

Mining Unit Online Component

Consolidation Unit Snapshots

Characterization Unit

Offline Component

Characterstics of dgp

Fig. 5.2. Architecture of DGP-CHAR System

The mining unit continuously mines the data stream to discover knowledge units which are considered to be at the lowest level of abstraction. Let K0i denote the pattern mined at time ti , at the 0th level of abstraction. These units are appropriately consolidated to capture changes and to produce knowledge units at a higher level of abstraction which are stored as snapshots. Requirement of the dgp to be analyzed at multiple abstraction levels motivates consolidation of Ki s. The operators in the characterization unit apply analytic methods on the preserved snapshots, to detect changes and draw conclusions about the characteristics of the underlying dgp over the speciﬁed time period. 5.3.1

Mining Unit

The mining unit of the DGP-CHAR system implements an appropriate stream mining algorithm. Stream mining algorithms are designed to work in real time

88

V. Bhatnagar and S. Kochhar

and in limited memory [5, 15]. They use diﬀerent window models to give importance to the recent data and discount the eﬀect of older data [2]. A large number of algorithms for frequent itemset mining [27], clustering [3, 10] and classiﬁcation [28] in data stream environment are available at [22]. The mining unit interacts with the focusing unit and periodically delivers the discovered patterns that are of interest. The focused search for patterns is both necessary and suﬃcient since monitoring of all discovered patterns is neither feasible nor desirable. The mining unit functions as a sampler which periodically mines the stream, eﬀectively producing a sample of discovered patterns (K0i ). These K0i s are fed to the consolidation unit for subsequent processing. 5.3.2

Consolidation Unit

Consolidation of the mined knowledge is an important step in characterization process. It facilitates comparative analysis of the Ki s at diﬀerent abstraction levels for inferencing the dgp characteristics. The consolidation unit captures changes in the discovered patterns by deriving change metrics at diﬀerent levels of abstraction. The derived metrics may capture the changes either implicitly or explicitly. The consolidation unit eﬀectively implements a change model, which is the basis of capturing changes and derivation of change metrics. The change model is deﬁned in the context of some speciﬁc characteristic(s) of the dgp and speciﬁes the type of changes that can be detected in this context. A rich model enables detection of changes of multiple types. A business house may be required to make predictions over a ﬁnancial year, half yearly, quarterly or monthly periods depending on the managerial level involved. Such requirements necessitate characterization of dgp at multiple time granularities by detection of changes at multiple levels of abstraction. Higher levels of abstraction highlight the features of the dgp in the long term, while the lower levels of abstraction indicate those in the short term. A higher level of abstraction is materialized by consolidating the metrics at a lower level. The idea is similar to drill-down/roll-up operations in data warehouse, with the distinction that diﬀerent levels of consolidation (abstraction) are now studied at user deﬁned granularities instead of those speciﬁed by the designer. The lowest level of abstraction at which the changes can be detected is at the level of mined results. The mined results K0t1 , K0t2 , . . ., the temporal units of knowledge at time instances t1 , t2 , . . . , respectively, are consolidated as per the change model over a time window w1 of size s1 , to yield knowledge discovered at one higher level of abstraction. In general, consolidation of knowledge discovered at level (i − 1) of abstraction over a time window wi of size si , yields knowledge unit Ki at level i of abstraction. Suitably designed consolidations at an abstraction level i yield change metrics of order i. Deﬁnition 1. A consolidation window wl at level l is a time window over which the discovered knowledge at abstraction level l − 1 is consolidated to yield discovered knowledge at abstraction level l.

5 Towards Characterization of the Data Generation Process

t0 t1 t2 t3 t4 t5 t6 t 7 t8 t9

DGP

k1 k2 k3 k4 k5 k6 k7 k8 k9

l=0

0

0

0

0

0

2

1

k1

k1 1 k2

0

0

0

0

3

k1

89

l 1= 1, s 1 = 3 l 2= 2, s 2 = 2

1 0 0 1 1 0

Fig. 5.3. Consolidation at level 0 and 1, with windows of sizes s1 = 3 and s2 = 2 respectively

Consolidation window wl is deﬁned to be of size sl , if sl temporal units of Kl−1 are consolidated to yield Kl (Figure 5.3). Deﬁnition 2. A consolidation function Cl for level l of abstraction with a consolidation window of size sl , combines sl knowledge units at level l-1 to yield a knowledge unit at level l (change metrics of order l) x+sl −1 x+1 x , Kl−1 , . . . Kl−1 ) −→ Kly , (y > x + sl − 1). i.e. Cl (sl , Kl−1 Kly is the derived change metric that expresses the extent of change in the lower order knowledge units during time period [tx , tx+sl −1 ] and is used for characterization of dgp at abstraction level l. It is interesting to note that the syntactic form of the Kl may be diﬀerent from that of Kl−1 . It may be meaningful to design more than one consolidation function at the same abstraction level to serve diﬀerent semantic requirements. Consequently, two diﬀerent consolidation functions at the same level may yield change metrics in varied forms. For instance to capture changes in cluster density and cluster orientation, diﬀerent consolidation functions need to be designed and the changes need to be expressed diﬀerently. Since semantics play an important role in design of consolidation functions, change model must integrate both objective and subjective aspects. 5.3.3

Oﬄine Component

The oﬄine component is modeled as a characterization unit that implements a characterization function and a set of operators designed for analyzing Ki s at different levels of abstraction. Both, the operators and the characterization function work upon the knowledge units stored as snapshots. The operators provide the desired view of the derived metrics, while the characterization function provides mapping to the select feature of the dgp, in order to enhance its understanding. Characterization Function The characteristics of dgp over a time span [ti , tj ] are a function of detected changes between the knowledge units, which are implicitly captured by the

90

V. Bhatnagar and S. Kochhar

snapshots. The objective of the characterization function Θl ([ti , tj ] is to map the detected changes to a set of suitable linguistic label(s) describing the characteristics of dgp e.g. consistent, unexpected. A generic characterizing function is deﬁned as follows. Deﬁnition 3. A characterization function Θl (ti , tj ) −→ C for level l of abstraction is a mapping based on the changes between knowledge units of comparable level generated during the time interval [ti , tj ]. C is a user specified set of linguistic labels. Since the snapshots comprise knowledge units at diﬀerent level of abstraction, a characterization function is required for each level of abstraction, which helps to understand the underlying semantic diﬀerences between the changes at diﬀerent levels of abstraction. Characterization is an area which is complex and highly prone to user subjectivity and requires a function that integrates both objective and subjective measures of interestingness. While the consolidation of change metrics for characterization is based on an objective function, the mapping to C captures user subjectivity. Operators Operators in the characterization unit support operations over the stored snapshots to facilitate analysis of change metrics over multiple time granularities from more than one perspective. Such analysis is useful not only for determining the pace of changes, but also for corroborating and reasoning the characteristics of dgp. An example set of operators is presented in Section 5.5.2.

5.4 Modeling Support Changes of Streaming Itemsets We instantiate the DGP-CHAR system shown in Figure 5.2 with a system designed to monitor changes in the support behavior of streaming itemsets. The proposed SI-CHAR system aims to characterize the consistency of the dgp based on the support behavior of the streaming itemsets. The change model implemented by the consolidation unit is based on one of our previous works [9]. We describe the online component describing the change model in this section. The oﬄine component is described in the next section. 5.4.1

I-Filter

The I-Filter captures user subjectivity and facilitates monitoring of only selected items in the selected support ranges, thereby leading to online data reduction and permitting ﬁne grained monitoring control. Since it is neither feasible nor desirable to monitor the support changes in all itemsets, the I-Filter allows a user to specify the monitoring space MS as a set of item-sets that the user wishes to monitor. Initially the user may select items

5 Towards Characterization of the Data Generation Process

91

User Focusing information

I−Filter

Stream Application

for Focusing Data Stream

DII−FP Tree Algo for Mining Support Statistics

Online Component

Consolidator Snapshots

Consistency Evaluator

Offline Component

Inferences about dgp consitency

Fig. 5.4. The SI-CHAR System

based on his requirement, experience or domain knowledge. Subsequently, the feedback about the behavior of item-sets provided by the system may inﬂuence user’s decision to update MS [8]. In most applications diﬀerent support ranges are more meaningful for diﬀerent categories of monitored item-sets, rather than discrete support thresholds. This is an extension of the idea of multiple minimum supports [31]. The I-Filter allows partitioning of the support space [0, 1] into p + 1 partitions using p support thresholds given by the user. 5.4.2

Mining Unit

The mining unit of SI-CHAR system implements DII-FP tree algorithm which is a variation of FP-tree algorithm to mine interesting itemsets. Since the set of interesting itemsets may change with time, the algorithm constantly interacts with I-Filter to mine for support of only those itemsets which are present in M s . Care is taken to decay the counts of itemsets which have not been seen in the data stream recently. Due to space constraints the details of algorithm are not provided here. Please refer to [9] for details. 5.4.3

Consolidation Unit

The consolidator accepts the support of interesting itemsets as input from the mining unit. Recall that the input from the mining unit constitutes knowledge units at 0th level of abstraction. The consolidation function restructures these knowledge units in accordance with the change model to produce change metrics of varying orders. In general, a change metric of order i corresponds to a suitably consolidated knowledge unit at abstraction level i. We ﬁrst describe the change model and then present the consolidation functions deﬁned by this model.

92

V. Bhatnagar and S. Kochhar

Change Model The change model is based on the premise that the overall consistency of the dgp is determined by the consistency of support behavior of the monitored itemsets (Ms ). Consistency of support of an itemset is captured by ascertaining the rate and quantum of change in its support over a speciﬁed time period. The support space of [0, 1] of an itemset is partitioned into p partitions in accordance with the support thresholds speciﬁed by user. The number of partitions deﬁne the granularity of observation. Let P i =[Sli ,Shi [ denote the ith partition where Sli and Shi denote lower and higher support thresholds bounding the partition.

*

t2

t1

*

t3 t5

P1

P2

0

P3

*

* t4

* Pk

1

Support Fig. 5.5. Support behavior of an itemset in partitioned support space for time interval [t1 , t5 ]

The M embership of an item-set x with support stx at time t is deﬁned as a mapping from [0, 1] −→ {1, . . . , p}: Mxt = i i|Sli ≤ stx < Shi (5.1) where i denotes a partition. The change model detects changes in support of itemsets by tracking the movement of itemsets across partitions. Note that Mxt is a restructured form of stx and hence both represent knowledge units at abstraction level 0. A transition of an item-set x at time t is deﬁned as a change in its membership with respect to that at time t − 1. 0 if Mxt = Mxt−1 t (5.2) Tx = 1 otherwise Consolidation Functions We now present the consolidation functions for the change model described above. A time window called p-window is used for consolidation of support changes at abstraction level (order) one and a time window called s-window is used for consolidation at abstraction level two. An s-window encompasses one or more p-windows as per the user desired time granularity.

5 Towards Characterization of the Data Generation Process

93

1. Persistence Factor Persistence Factor is the change metric of order one, derived by consolidating the support value of an itemset over a p-window. Although each individual transition is both meaningful and interesting, sometimes non signiﬁcant changes in the support of an itemset may cause false alerts. The membership of an item-set over ap-windowof size sl is observed, before concluding about its support behavior. Consolidation of membership information during the p-window gives a measure of persistence of an item set in a partition. Deﬁnition 4. Persistence factor (PFxp ) of an item-set x for a support partition p quantifies the extent of its existence in that support partition during a p-window. It is computed as follows: ( no. times xs1existed in p ). Example 1. Figure 5.6 shows the membership of an item-set x at ﬁve time instances during p-window with s1 = 5. Using these supports as K01 , K02 , . . . , K05 , PF for partitions 3 and 2 are computed respectively as follows: P Fx3 = 3/5 and P Fx2 = 2/5. For all other partitions PF of x is 0/5.

Mining Time

Partition No.

1 3

2

3

2

3

4 2

5 3

Fig. 5.6. Computation of PF of an item-set x

PF of an item-set implicitly represents the extent of change in support of an itemset in a particular partition i.e. support range High value of PF in a partition indicates little variation in the value of support. This metric however is purely objective and does not capture user subjectivity. 2. Item-set Spread Spread of an item-set x, a change metric of order two, is the number of support partitions in which its support was found to fall during an s-window. It acts as an indicator of the range of support variation during the s-window. A higher value of Spread indicates higher extent of variation in the support of an item-set. Spread of an itemset x in a s-window at time t is computed as follows: p 2 ( P Fxj ) Sptx = maxsi=1 j=1

(5.3)

The term counts the number of partitions with non-zero PF in a pwindow. Spread of an item-set x is the maximum value of the counts over all

94

V. Bhatnagar and S. Kochhar

p-windows in the s-window. Thus for an item x, if s2 = 1 (i.e. p-window = s-window), Sptx = 2 (Figure 5.6), indicating that it was present in only two partitions during the s-window. 3. Item-set State The State of an item-set, which is a change metric of order two, is also derived from the PF of an item-set in a partition. The State of an item-set is an expression of the stability of its support in a partition during a swindow. State Stpx of an item-set x in partition p is a mapping from [0, 1] → {M iss, Rare, T ransient, Consistent, Concrete}, deﬁned as follows: ⎫ ⎧ Miss Avg(P Fxp ) ≤ θmiss ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ Rare θmiss ≤ Avg(P Fxp ) < θrare ⎬ ⎨ p p (5.4) Stx = Transient θrare ≤ Avg(P Fx ) < θtrans ⎪ ⎪ p ⎪ ⎪ Consistent θ ≤ Avg(P F ) < θ ⎪ ⎪ trans cons ⎪ x ⎪ ⎭ ⎩ Concrete Avg(P Fxp ) ≥ θcons where 0 ≤ θmiss ≤ θrare ≤ θtrans ≤ θcons ≤ 1 are the cut points speciﬁed by the user. It follows from the deﬁnition that while State is computed objectively, the mapping function captures the user subjectivity. 4. Item-set Transition Rate The Transition Rate is also a change metric of order two which characterizes an item-set with respect to the average number of inter-partition movements it makes over the s-window. Transition Rate T Rtx of an item-set x for swindow at time t is computed as follows: s2 s1 j i=1 j=1 Tx t T Rx = (5.5) s1 ∗ s2 where Txj is the transition of itemset x at time j. The inner summation counts the transitions in a p-window, while the outer summation consolidates these counts over the s-window. Please note that though State, Spread and Transition Rate represent knowledge units at abstraction level two, their syntactic forms are diﬀerent (See section 5.3.2).

5.5 Characterizing Consistency of Streaming Itemsets The consistency evaluator of SI-CHAR system draws inferences about the consistency of the support behavior of monitored itemsets and dgp, based on the snapshots of change metrics stored by the consolidator. The characterization function implemented by the consistency evaluator ascertains consistency of the dgp underlying data stream by quantifying the extent of change in the support of itemsets in M s . A set of operators to enable semantic interpretation of the change metrics from more than one perspective.

5 Towards Characterization of the Data Generation Process

5.5.1

95

Characterization Function

The characterization function applies systematic consolidation on the modeled changes stored as snapshots to make a statement about consistency of the underlying data generation process during a speciﬁed time period. It provides a long term perspective of the relative stability and consistency of the support behaviors. Since the notion of consistency is highly subjective with respect to the application domain being monitored, the metric can be computed in more than one of the ways. We deﬁne below two characterization functions which compute a metric called Consistency Quotient (CQ) to quantify the overall consistency of the dgp during a speciﬁed time period. CQ can be computed based in the Spread and State as given below. CQ based on Spread Computation of CQ based on Spread is done on the premise that the overall consistency of the dgp is the aggregated behavior of the consistency of the individual itemsets being monitored. For user speciﬁed M S and p partitions in the support space, the consistency of the dgp is a function of Spread of the itemsets. Since higher Spread over a time period indicates higher volatility, item-sets with lower value of Spread contribute more to the consistency of the support behavior of the data stream. Let ci be the count of the item-sets (∈ M S ) with Spread i (1 ≤ i ≤ p). Consistency Quotient based on Spread of the item-sets is deﬁned as follows: p 1 i=1 i ∗ ci CQspread [ti , tj ] = (5.6) |M S | w∈[ti ,tj ]

where w denotes an s-window and the outer summation consolidates the Spread of itemsets for all windows during the time [ti , tj ]. Since Spread gives an idea of the "no-change" support behavior of the itemsets, a non-existent (support = 0) itemset in the stream contributes as much to the consistency as an itemset with support= 100%. Though consistent absence can be an interesting piece of knowledge in some situations, CQspread does not indicate the quantum of support of the monitored itemsets. CQ based on State CQstate is the a proportion of itemsets that exhibit a concrete state during all the s-windows constituting time [ti , tj ]. A concrete state is indicative of highest stability of an item-set irrespective of the partition. Thus the proportion of itemsets with concrete state is indicative of the stability of the support behavior of item-sets. Consistency Quotient based on State of the item-sets is computed as: p nc i=1 i (5.7) CQstate = |M S | w∈[ti ,tj ]

96

V. Bhatnagar and S. Kochhar

where nci is the number of concrete item-sets in partition i. The numerator counts number of itemsets with concrete state for each window during the speciﬁed time period and the denominator normalizes the sum so that 0 ≤ CQstate ≤ 1. 5.5.2

Operators

The consistency evaluator is augmented with a set of operators to provide an interface for semantic interpretation of the change metrics recorded by the consolidator. There can be other approaches to achieve the objective including importing the snapshots to a relational database and using SQL to query the changes. However, we consciously selected the operator based approach because of ﬂexibility, expandability and ease of semantic interpretation. In case the change model is expanded or modiﬁed the set of operators can be suitably updated. We describe the operators to query Membership , State, Spread and Transition rate. 1. Membership Operators (M EM ) M EM operator analyzes Persistence Factor metric (snapshots) to provide membership information over a time interval [ti ..tk ]. Two variants of the operator are proposed to facilitate two-dimensional analysis. The IMEM operator delivers support behavior of an individual itemset to enable study of its varying support behavior, while the PMEM operator facilitates characterization of a support range w.r.t. the itemsets contained in it. The IMEM operator allows a user to query the presence of a particular itemset in various partitions. It is designed to retrieve properties of an individual itemset to enable study of its time variant support behavior. The IMEM operator takes as arguments the name of an itemset and a time interval and returns a list of all the partitions to which itemset I belonged during time interval [ti ..tk ]. Thus IMEM (I, [ti ..tk ]) gives an idea of the range of variation of the itemset support during the speciﬁed time and is an extension of the Spread metric. The PMEM operator allows a user to query the characteristics of a support range w.r.t. the itemsets present in it. The PMEM operator takes as arguments the partition and a time interval and outputs a list of all the itemsets which were a member of partition p during time interval [ti ..tk ]. Thus PMEM (p, [ti ..tk ]) facilitates focusing on one support range w.r.t. the itemsets in the stream during that period. The State Operator The State operator analyzes snapshots of State change metric to provide information about the consistency of support behavior of an itemset over a time interval [ti ..tk ]. Two variants of the State operator viz. the IState operator and the PState operator are proposed. While the IState operator retrieves consistency information about an individual itemset’s support behavior, the PState operator permits

5 Towards Characterization of the Data Generation Process

97

inferencing of the characteristics of a partition. w.r.t. State of itemsets contained in it. The IState operator accepts as input an itemset I, a partition p and a time interval and outputs a list of linguistic labels that describe the consistency of itemset I’s support behavior w.r.t. partition p during the time interval [ti ..tk ]. The usage is: IState (I, P, [ti ..tk ]) The PState operator accepts as input a partition p and a time interval and lists State(s) of all the itemsets in partition p during the time interval [ti . . . tk ]. The usage is: Pstate (p, [ti ..tk ]) The T R Operator The T R operator allows a user to query the number of transitions made by an itemset during a speciﬁed time period. It accepts as arguments the queried itemset I and a time interval. The output is the Transition Rate of itemset I during time interval [ti ..tk ]. The usage is: T R(I, [ti ..tk ])

5.6 Monitoring Usage of E-Resources in Delhi University Library System In this section we present an application of SI-CHAR system to study the usage patterns of e-resources subscribed by the Delhi University Library System (DULS). DULS, with over 30 libraries in its fold, oﬀers services through its campus wide network to the academic community of Delhi University including its teachers, students and research scholars [17]. The library subscribes a large number of electronic databases, including the Reference Sources, Full Text Sources and Bibliographic Databases. The motivation for the study was to identify the resources that have consistent low usage and those which have consistent high usage. This knowledge can be used by the library system to generate recommendations for contracts with publishers and digital libraries in order to optimize the ﬁnancial liabilities of the library. 5.6.1

Data Preparation

The web access log containing the click stream data for Jan. - Feb. 2008, was obtained from an Apache server in Delhi University Computer Centre (DUCC). The log, originally in DansGuardian format, was preprocessed and transformed to a transactional format as described below. The records of the webclicks pertaining only to the subscribed e-resources were selected from the web access log and the attributes Date, Time, Department id and e-resource were extracted. The log ﬁle was divided into sessions, each session corresponding to the e-resources accessed by one department in an hour. Each session is a record of e-resources that have been accessed by a particular department in one hour. The transactions were compressed to remove duplicate instances of e-resources. A total of 182652 records were streamed in for study.

98

V. Bhatnagar and S. Kochhar

5.6.2

Results

We present the results with a view to demonstrate the claims made in Section 5.4. We show that the change metrics faithfully capture the variations in support behavior of selected e-resources. This can be veriﬁed from the support graphs of the corresponding items shown in Figures 5.7 and 5.8. The following parameters were supplied through a conﬁguration ﬁle: i) Size of p-window = 2, size of s-window = 3 ii) List of 13 e-resources constituting the Monitoring Space M S iii) Support thresholds: {0.1, 0.3, 0.4, 0.5, 0.6, 0.7 } partitioning the support space into seven partitions iv) Thresholds for State computation: θmiss = 0.1, θrare = 0.3, θtrans = 0.6, θcons = 0.8. Because of lack of space, we present the results for only two e-resources for time period captured by ﬁve s-windows. The ﬁrst e-resource, Proquest shows consistent high support while the second one, Springerlink shows variability as shown by Figures 5.7 and 5.8 respectively. These plot the support of the respective items (mined by the mining unit) against time for ﬁve s-windows. Tables 5.1 and 5.2 depict the change metrics retrieved from the snapshots using the operators presented in Section 5.5.

Fig. 5.7. Support of Proquest

Fig. 5.8. Support of Springerlink

Table 5.1 shows the State, Spread and Transition Rate of Proquest database vindicating the support behavior depicted by Figure 5.7. The item is restricted to one partition (Spread = 1), never transits (TR = 0) and maintains a Concrete State in the partition P7 (support range: 0.7-1.0) throughout the observation period. Variations of the support graph for Springer database (Figure 5.8) are captured by Table 5.2. The Spread indicates presence of the item in maximum two partitions, signifying variability in three out of ﬁve windows. Of the three time periods in which the Spread of e-resource is two, the maximum transition in support behavior is seen during time window W 1. Based on the observations during the limited time period of study it can be concluded that the library should deﬁnitely continue subscription of Proquest

5 Towards Characterization of the Data Generation Process

99

Table 5.1. Change metrics for Proquest database s-window W1 W2 W3 W4 W5

P1

P2

P3

Miss Miss Miss Miss Miss

Miss Miss Miss Miss Miss

Miss Miss Miss Miss Miss

State P4 P5 Miss Miss Miss Miss Miss

Miss Miss Miss Miss Miss

Spread TR P6

P7

Miss Miss Miss Miss Miss

Concrete Concrete Concrete Concrete Concrete

1 1 1 1 1

0 0 0 0 0

Table 5.2. Change metrics for Springerlink database s-window W1 W2 W3 W4 W5

State P1

P2

Miss Miss Miss Miss Miss

Miss Miss Miss Miss Miss

P3

P4

Spread TR P5

P6

Miss Consistent Miss Miss Miss Concrete Miss Miss Miss Miss Concrete Miss Miss Miss Consistent Rare Miss Miss Transient Transient

P7 Rare Miss Miss Miss Miss

2 1 1 2 2

0.5 0 0 0.16 0.16

database. The conﬁdence for concluding the behavior of Springer database however is low because of higher variation in its support. Library may launch an educational campaign for promotion of the database. The study also gives conﬁdence that a more involved study, for e.g. a study over a whole academic session is expected to help in improved decision making and planning for optimum utilization of resources.

5.7 Monitoring Traﬃc Accident Data As a second application, we applied the SI-CHAR framework on a data set of traﬃc accidents that involved injured or fatal casualties on a public road in Belgium for the period 1991-2000. The data for study was obtained from Fimi03 website [20]. The data set contains 340,184 records with 572 diﬀerent attribute values. Since there was no explicit chronological order of the records in the dataset, we synthetically generated timestamps as per the order of records. 5.7.1

Focusing and Conﬁguration Settings

The following parameters for focusing were supplied through a conﬁguration ﬁle: i) M S , the monitoring Space was constituted by the following attributes: 1. Part of the day: This attribute records in which of the following ﬁve time zones viz. morning hour (10-12 hours), afternoon (13 - 15 hours), evening

100

V. Bhatnagar and S. Kochhar

rush hour (16 - 18 hours), evening (19 - 21 hours), night (22 - 6 hours) the accident occurred. 2. Attribute Road Conditions: This attribute, with the following ﬁve attribute values, tracks the type of road surface on which an accident occurred: dry road surface, wet road surface, snow on road surface, clean road surface, or dirty road surface. 3. Consequences of victim : This attribute notes down the type of cardinality. The valid attribute values are: dead victim, seriously injured victim or lightly injured victim. 4. sex of the victim : This attribute records whether the victim was a male or female. ii) Size of p-window = 2 time units, size of s-window = 3 p-windows iii) Support thresholds: {0.2, 0.4, 0.6, 0.8} iv) Thresholds for State computation: θmiss = 0.1, θrare = 0.3, θtrans = 0.6, θcons = 0.8. 5.7.2

Results

This section presents the change metrics and the resulting conclusions for two pairs of itemsets: i) {Dead, Male} and {Dead, Female}, ii) {Morning_rush_hour, Seriously_injured} and {Evening_rush_hour, Seriously_injured}. Results for only these itemsets are presented because of limited space and interesting contrasts in their support behavior. The support graphs of itemsets {Dead, Male} and {Dead, Female} are presented in Figures 5.9 and 5.10 and the Tables 5.3 and 5.4 present their respective change metrics. Table 5.3 shows that the itemset {Dead, Male} is Miss from partitions P3, P4 and P5 during the entire period of observation. Towards the end of observation period, the support of the itemset stabilizes in the range [0.2-0.4]. This can be veriﬁed from the Figure 5.9. Figure 5.10 shows more variability in the support of the itemset {Dead, Female}in the range [0.2-0.4] which is captured by Table 5.4. The number of female deaths, initially high, became low during window W 2, similar to the trend observed in {Dead, males}. The number of female casualties rose consistently in next time window making the appearance of itemset {Dead, Female} in partition P2 Rare. It may be noted that this is the time window with maximum Transition Rate. The subsequent time window witnesses a fall in the support of itemset which remains stable for the remaining monitored time. Overall, the itemset {Dead, Female} is restricted to partitions P2 and P3 indicating a higher number of deaths in comparison with men. Interestingly, the Transition Rate for both males and females is high in the time window W 3 which may give a pointer for investigation of external factors (e.g. road conditions) due to which changes in the dgp have come about.

5 Towards Characterization of the Data Generation Process

Fig. 5.9. Support of itemset {Dead, Male}

101

Fig. 5.10. Support of itemset {Dead, Female}

Table 5.3. Change metrics for itemset {Dead, Male} s-window

State P2 P3

P1 W1 W2 W3 W4 W5

Transient Concrete Rare Miss Miss

Transient Miss Consistent Concrete Concrete

Miss Miss Miss Miss Miss

Spread TR P4

P5

Miss Miss Miss Miss Miss

Miss Miss Miss Miss Miss

2 1 2 1 1

0.13 0 0.13 0 0

Table 5.4. Change metrics for itemset {Dead, Female} s-window P1 W1 W2 W3 W4 W5

P2

State P3

Spread TR P4

P5

Miss Miss Concrete Miss Miss Miss Concrete Miss Miss Miss Miss Rare Consistent Miss Miss Miss Concrete Miss Miss Miss Miss Concrete Miss Miss Miss

1 1 2 1 1

0 0 0.13 0 0

The Figures 5.11 and 5.12 present the support graphs for itemsets {Morning_rush_hour, Seriously_injured} and {Evening_rush_hour, Seriously_ injured}, while Tables 5.5 and 5.6 show the captured change metrics for these itemsets. The itemset {Morning_rush_hour, Seriously_injured} demonstrates lesser volatility in its support during the entire observation period. This is evident from the value of Spread and Transition Rate(Table 5.5). Both indicate that the

102

V. Bhatnagar and S. Kochhar

Fig. 5.11. Support of itemset {Morn- Fig. 5.12. Support of itemset {Evening_ ing_rush_hour, Seriously_injured} rush_hour, Seriously_injured} Table 5.5. Change metrics captured by the online component for itemset {Morning_rush_hour, Seriously_injured} s-window W1 W2 W3 W4 W5

P1

P2

Miss Miss Miss Miss Miss

Miss Miss Miss Miss Miss

State P3

Spread TR P4

P5

Concrete Miss Miss Concrete Miss Miss Concrete Miss Miss Transient Transient Miss Concrete Miss Miss

1 1 1 2 1

0 0 0 0.33 0

Table 5.6. Change metrics captured by the online component for itemset {Evening_rush_hour, Seriously_injured} s-window W1 W2 W3 W4 W5

P1

P2

Miss Miss Miss Miss Miss

Miss Miss Miss Miss Miss

P3

State P4

Spread TR P5

Miss Concrete Miss Miss Rare Consistent Miss Miss Concrete Miss Miss Concrete Miss Miss Concrete

1 2 1 1 1

0 0.13 0 0 0

number of seriously injured people during the Morning rush hour varies only in the time window W 4, when it becomes Transient in partitions P3 and P4. The incidence of seriously injured people in evening rush hour is consistently higher than that of people seriously injured in morning rush hour. The inference can be made on the basis of Table 5.6 which shows the presence of itemset {Evening_rush_hour, Seriously_injured} in partitions P4 and P5. The Spread value of two during window W 2 indicates that itemset support was varying. The

5 Towards Characterization of the Data Generation Process

103

State complements this information by showing that the presence of itemset is Rare in partition P4 and Consistent in the partition P5. The subsequent time windows reﬂect that the number of people seriously injured in the evening rush hour remains stable. It can be concluded that a detailed study of this type will be useful in identifying casual factors, thereby improving the traﬃc management and reducing the road risk.

5.8 Conclusion and Future Directions This chapter outlines the major tasks that need to be addressed for deducing characteristics of the data generation process. An architecture that caters to these tasks and can serve as a basic template for the process of dgp characterization is also proposed. Since the characterization process is expected to be complex, we argue for characterizing the dgp w.r.t. a speciﬁc characteristic. Further, it is also important to take user subjectivity into account for the endeavor to be useful. There is tremendous scope of dgp characterization in e-auditing and egovernance applications. However, design of functions for modeling changes, consolidation and characterization are crucial and theory needs to be developed for design of such functions for each knowledge type.

Acknowledgment We gratefully acknowledge the grant (Grant number-Dean(R)/R&D/2007/PhIII/382) provided by Delhi University for carrying out this research. We are also thankful to the Delhi University Computer Centre and Fimi [26] for the data. Guidance provided by S.C.Jindal, Librarian, Central Science Library, University of Delhi is acknowledged with gratitude. Comments on the draft by S.K.Gupta, IIT Delhi and Sharanjit Kaur, University of Delhi helped us to improve the manuscript.

References 1. Aggarwal, C.C.: A framework for diagnosing changes in evolving data streams. In: Proceedings of the 2003 ACM SIGMOD International Conference on Management of Data, pp. 575–586. ACM Press, New York (2003) 2. Aggarwal, C.C.: Data Streams: Models and Algorithms (Advances in Database Systems). Springer, New York (2006) 3. Aggarwal, C.C., Han, J., Wang, J., Yu, P.S.: A framework for clustering evolving data streams. In: Proceedings Of International Conference On Very Large Data Bases (2003)

104

V. Bhatnagar and S. Kochhar

4. Tsymbal, A.: The Problem of Concept Drift: Deﬁnition and Related Work, www.cs.tcd.ie/publications/tech-report/reports.04/ TCD-CS-2004-15.pdf 5. Babcock, B., Babu, S., Datar, M., Motwani, R., Widom, J.: Models and issues in data stream systems. In: PODS 2002: Proceedings of the twenty-ﬁrst ACM SIGMOD-SIGACT-SIGART symposium on Principles of database systems, pp. 1–16. ACM, New York (2002) 6. JohannesGehrke Ben-David, S., Kifer, D.: Detecting change in data streams. In: Proceedings Of International Conference On Very Large Data Bases (2004) 7. Bhatnagar, V.: Intension Mining: A New Approach to Knowledge Discovery in Databases. PhD thesis, JMI, New Delhi, India (2001) 8. Bhatnagar, V., Kochhar, S.: User subjectivity in change modeling of streaming itemsets. In: Proceedings Of 1st International Conference On Advanced Data Mining Applications (July 2005) 9. Bhatnagar, V., Kochhar, S.: Modeling support changes in streaming item sets. International Journal of Systems Science 37(13/20), 879–891 (2006) 10. Bhatnagar, V., Kaur, S.: Exclusive and complete clustering of streams. In: Wagner, R., Revell, N., Pernul, G. (eds.) DEXA 2007. LNCS, vol. 4653, pp. 629–638. Springer, Heidelberg (2007) 11. Bhatnagar, V., Kochhar, S.: Beyond mining: Characterizing the data generation process. In: Proceedings of Seventh International Conference on Intelligent Systems Design and Applications, ISDA 2007, Rio de Janeiro, Brazil, pp. 491–496 (2007) 12. Baron, S., Spiliopoulou, M.: Monitoring the evolution of web usage patterns. In: EWMF, pp. 181–200 (2003) 13. Chen, M.-C., Chiu, A.-L., Chang, H.-H.: Mining changes in customer behavior in retail marketing. Expert System Appllications 28(4), 773–781 (2005) 14. Cormode, G., Muthukrishnan, S.: What is new: Finding signiﬁcant diﬀerences in network data streams. In: Proceedings of INFOCOM (2004) 15. Domingos, P., Hulten, G.: Catching up with the data: Research issues in mining data streams. In: ACM SIGMOD Workshop on Research Issues in Data Mining and Knowledge Discovery (2001) 16. Dong, G., Han, J., Lakshmanan, L.V.S., Pei, J., Wang, H., Yu, P.S.: Online mining of changes from data streams: Research problems and preliminary results. In: Proceedings of ACM SIGMOD (2003) 17. Delhi university library system, http://crl.du.ac.in 18. Fan, W.: Streamminer: A classiﬁer ensemble-based engine to mine concept drifting data streams. In: Proceedings of International Conference on Very Large Data Bases (2004) 19. Fan, W., Huang, Y., Wang, H., Yu, P.S.: Active mining of data streams. In: Proceedings of International Conference SIAM (2004) 20. Repository of the 1st international workshop on frequent itemset mining implementations (ﬁmi 2003). In: IEEE ICDM 2003, Melbourne, Florida, USA (2003)

5 Towards Characterization of the Data Generation Process

105

21. Fayyad, U.M., Piatetsky-Shapiro, G., Smyth, P.: From data mining to knowledge discovery: An overview. In: Advances in Knowledge Discovery and Data Mining, pp. 1–34 (1996) 22. Gaber, M.M.: Mining data streams bibliography, http://www.csse. monash.edu.au/~mgaber/WResources.htm 23. Gupta, S.K., Bhatnagar, V., Wasan, S.K.: Architecture for knowledge discovery and knowledge management. Knowl. and Inf. Syst. 7(3), 310–336 (2005) 24. Ganti, V., Gehrke, J., Ramakrishnan, R.: Demon: Mining and monitoring evolving data. In: ICDE, pp. 439–448 (2000) 25. Ganti, V., Gehrke, J., Ramakrishnan, R., Loh, W.-Y.: Focus: A framework for measuring diﬀerences in data characterstics. In: Proceedings of PODS (1999) 26. Geurts, K., Wets, G., Brijs, T., Vanhoof, K.: Proﬁling high frequency accident locations using association rules (2003) 27. Han, J., Pei, J., Yin, Y.: Mining frequent patterns without candidate generation. In: Proceedings of International. Conf. SIGMOD 2000 (May 2000) 28. Hulten, G., Spencer, L., Domingos, P.: Mining time-changing data streams. In: Proceedings of the ACM SIGKDD (2001) 29. Medhat, M., Gama, G.J. (eds.): Learning from Data Streams. Springer, Heidelberg (2007) 30. Liu, B., Hsu, W., Han, H.-S., Xia, Y.: Mining changes for real-life applications. In: Kambayashi, Y., Mohania, M., Tjoa, A.M. (eds.) DaWaK 2000, vol. 1874, p. 337. Springer, Heidelberg (2000) 31. Liu, B., Hsu, W., Ma, Y.: Mining association rules with multiple minimum supports. In: KDD 1999: Proceedings Of The Fifth ACM SIGKDD International Conference On Knowledge Discovery And Data Mining, pp. 337–341. ACM, New York (1999) 32. Padmanabhan, B.: Unexpectedness as a measure of interestingness in knowledge discovery, IS-97-06 (1997) 33. Piatetsky-Shapiro, G., Mathens, C.J.: The interestingness of deviations. In: Proceedings of the AAAI 1994 Workshop on Knowledge Discovery in Databases, pp. 25–36 (1994) 34. Spiliopoulou, M., Ntoutsi, I., Theodoridis, Y., Schult, R.: Monic: Modeling and monitoring cluster transitions. In: KDD 2006: Proceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 706–711. ACM, New York (2006) 35. Silberschatz, A., Tuzhilin, A.: What makes patterns interesting in knowledge discovery systems. IEEE Trans. on Knowl. and Data Eng. 8(6), 970–974 (1996) 36. Wang, H., Fan, W., Yu, P.S., Han, J.: Mining concept drifting data streams using ensemble classiﬁers. In: Proceedings of the ACM SIGKDD (2003) 37. Wang, K., Zhou, S., Fu, A., Yu, J.: Mining changes of classiﬁcation by correspondence tracing. In: Proceedings of International SIAM Data Mining conf. (2003)

6 Data Mining Applied to the Electric Power Industry: Classification of Short-Circuit Faults in Transmission Lines Yomara Pires, Jeﬀerson Morais, Claudomir Cardoso, and Aldebaro Klautau Signal Processing Laboratory (LaPS) – Federal University of Pará (UFPA) CP 8619 – 66075-110 – Belém – PA – Brazil {yomara,jmorais,claudomir,aldebaro}@ufpa.br www.laps.ufpa.br

Data mining can play a fundamental role in modern power systems. However, the companies in this area still face several diﬃculties to beneﬁt from data mining. A major problem is to extract useful information from the currently available non-labeled digitized time series. This work focuses on automatic classiﬁcation of faults in transmission lines. These faults are responsible for the majority of the disturbances and cascading blackouts. To circumvent the current lack of labeled data, the Alternative Transients Program (ATP) simulator was used to create a public comprehensive labeled dataset. Results with diﬀerent preprocessing (e.g., wavelets) and learning algorithms (e.g., decision trees and neural networks) are presented, which indicate that neural networks outperform the other methods.

6.1 Introduction Data mining techniques have been used in the electric power industry for a variety of tasks. Illustrative examples are: protection systems [1], load forecast [2] and customer proﬁle analysis [3]. This work focuses on classiﬁcation of time series, which is important in the context of power quality, for example. In many countries, including Brazil, new regulations require the electric power industry to have strict control on the quality of their service, under the risk of being penalized with expensive ﬁnes. Therefore, power quality (PQ) [4, 5, 6] became an important issue. A PQ event is generally deﬁned as any disturbance or occurrence manifested in the voltage levels, current or in the frequency variations that results in fault or bad operation of consumers’ equipments [4]. To date, the electric power industry has a reasonably sophisticated logistics to acquire and store time series (waveforms) corresponding to PQ events. However, mining such data to infer, e.g. cause-eﬀect relations, is an incipient activity. One of the main problems is that the digitized time series are often stored without proper labels, which complicates the adoption of supervised learning [7,8,9]. For example, oscillography equipments [10] implement relatively simple algorithms that detect if the voltage waveforms (sinusoids) deviate from their nominal N. Nedjah et al. (Eds.): Innovative Applications in Data Mining, SCI 169, pp. 107–122. c Springer-Verlag Berlin Heidelberg 2009 springerlink.com

108

Y. Pires et al.

amplitude value. If the variation is larger than a threshold, the programmable trigger circuit starts storing the data, along with additional information such as date and time. In spite of storing the PQ event, typically there is no action to relate the event with additional data that could help, e.g., inferring the cause. Establishing this relation is a long term goal in the electric power industry. It requires a data warehouse to integrate their legacy SCADA (Supervisory Control and Data Acquisition) systems [11,12], which report time to the second and do not provide waveforms, with the new intelligent electronics devices (IEDs) such as digital fault recorder (DFRs) and digital relays, which can support sampling frequencies in the order of few tens of kHz and implement sophisticated relay algorithms [10]. Before this scenario becomes a reality, there is a urgent need to facilitate the computational intelligence community to get acquainted with the area, for being able to establish eﬃcient strategies to help specialists to label the time series. A strong motivation to this work is that electric power companies have a large amount of unlabeled oscillography data and want to mine it for establishing cause-eﬀect relations. This work describes basics concepts about data mining of power signal time series and a methodology adopted to circumvent the vicious circle: there is no algorithm to properly label the data because there is no labeled data for supervised learning. The strategy is based on narrowing the scope of the investigation and provide a solution to the automatic analysis of a particular and important class of causes of PQ events: faults in transmission lines. Studies showed that these faults were responsible for 70% of the disturbances and cascading blackouts [4, 13]. In a ﬁrst stage, the faults are simulated with the Alternative Transients Program (ATP) [14] and data mining techniques (preprocessing and machine learning algorithms) are used to train and test classiﬁers. ATP models have a long history of good reputation and are well-tuned to the actual system behavior. Hence, the artiﬁcially-generated data can be (hopefully) used for training sensible classiﬁers. In a second stage these classiﬁers will help pre-labeling the oscillography data, from which a subset is post-processed by specialists to correct/conﬁrm the labels, assess the procedure and allow reﬁnement through new iterations. This work presents the results of the ﬁrst stage, focusing on the experimental evaluation of front ends and learning algorithms. Another contribution is the development and free distribution of a dataset of labeled time series based on the framework proposed in [15]. Similarly to other time-series mining applications [16], investigations on PQ lack freely available and standardized benchmarks. The datasets of PQ events are often proprietary and, consequently, reproducing previously published results is impossible. The second contribution is a thorough review of the issues and evaluation of preprocessing and machine learning algorithms applied to fault classiﬁcation. The dataset and review will hopefully help computational intelligence experts to apply their algorithms to the PQ ﬁeld. It should be noticed that the task of detecting the deviation of a waveform parameter from its nominal value is called here trigger and distinguished from

6 Data Mining Applied to the Electric Power Industry

109

classification tasks. The trigger problem (see, e.g., [17]) consists in deciding whether or not a waveform parameter (such as frequency or amplitude) is too far (given a speciﬁed level of tolerance) from its nominal value. The trigger problem can be seen as a special (binary and relatively easy) case of outlier detection [18]. When a trigger procedure detects an event (a deviation), extra waveform parameters can be estimated. All three tasks, namely: classiﬁcation, trigger and waveform analysis, have been considered as part of data mining processes in the recent literature. The following example illustrates the reason for clearly distinguishing them. Two important PQ events are sags and swells, which correspond to a decrease and increase on the nominal amplitude, respectively [4]. Using a sliding window to iteratively segment the time series, it is possible to calculate the root meansquare (RMS) value corresponding to each window (time interval) [4]. The RMS time-evolution can help detecting sags and swells, which can then be organized in categories such as critical / non-critical (depending on the deviation from the nominal RMS) and short / long (depending on duration). Strictly, this is not classiﬁcation and depends only on the accuracy of the chosen parameter estimation (RMS value in this example). Having an estimate for the parameter, the PQ event is simply “categorized” according to established standard tables, which map the parameter ranges into categories. Trigger and waveform analysis typically do not require data mining techniques and, in this aspect, are diﬀerent from the fault classiﬁcation investigated in this work. This work is organized as follows. Section 6.2 describes the classiﬁcation of time series representing faults and deﬁnes notation. Section 6.3 discusses a public dataset with faults in transmission lines. Section 6.4 presents the adopted preprocessing and learning algorithms used in the experiments, with their associated baseline results. Section 6.5 presents the conclusions.

6.2 Classification of Time Series Representing Faults In the electric industry, a signiﬁcant volume of collected and stored data corresponds to time series. In this work, the time series correspond to faults, which are basically short-circuits in transmission lines. Most transmission systems use three phases: A, B and C. Hence, a short-circuit between phases A and B will be identiﬁed as “AB”. Considering the possibility of a short-circuit to “ground” (G), the task is to classify a time series into one among ten possibilities: AG, BG, CG, AB, AC, BC, ABC, ABG, ACG and BCG. The ABC and ABCG faults are typically not distinguished because in well-balanced circuits (or ATP simulations) there is no current ﬂow through the ground [19]. Algorithms to solve this classiﬁcation problem are used by digital fault recorders (DFRs), distance relays and other equipments (see, e.g., [10]). The signal capturing equipments are sometimes located at both endpoints of the transmission line. Most of them are capable of digitizing both voltage and current waveforms. It is assumed that a trigger circuit detects an anomaly and stores only the interval of interest - the fault and a pre-determined number of

110

Y. Pires et al.

samples before and after the fault. The trigger is out of the scope of the present work and the simulations assumed a perfect trigger algorithm, with the fault endpoints being directly obtained from the simulator. 6.2.1

Front Ends

This subsection describes the front end , the stage in charge of providing a suitable parametric representation to the time series. At some points the notation may look abusive, but there are many degrees of freedom when dealing with time series and a precise notation is necessary to avoid obscure points. Each fault is a variable-duration multivariate time-series. The n-th fault Xn in a dataset (oscillography records, for example) is represented by a Q × Tn matrix. A column xt of Xn , t = 1, . . . , Tn , is a multidimensional sample represented by a vector of Q elements and Tn is the number of frames. For example, this work adopts Q = 6 (voltage and current of phases A, B and C) in the experiments. In some situations [13], it is possible to obtain synchronized samples from both endpoints of a given line. In these cases the sample is an augmented vector with twice the dimension of the single endpoint scenario. For the previous example, the sample dimension for double endpoint measures would be Q = 12. A front end converts samples into features for further processing. An example of a modern front end algorithm is the wavelets decomposition [20]. Independent of the adopted parametric representation, a single sample typically does not carry enough information to allow performing reasonable decisions. Hence, it is useful to consider that a front end converts the matrix X into a matrix Z with dimension K × N , as depicted in Figure 6.1 (the processing is performed on Z, not X), where K is the number of features and N the number of feature vectors. The following subsections describe some popular front ends.

Fig. 6.1. The input and output matrices of the front end stage. Q and K are the dimension of the sample and feature vectors, respectively.

Framed raw front end A front end is called raw when it outputs features that correspond to values of the original samples, without any processing other than organizing the samples into a matrix Z. In the framed raw front end, this organization is obtained through an intermediate representation called frame. A frame F has dimension Q × L, where L is the number of samples called frame length and their concatenation ˆ = [F1 . . . FN ] is a matrix of dimension Q × LN , where N is the number of Z frames.

6 Data Mining Applied to the Electric Power Industry

111

The frames can overlap in time such that the frame shift S, i.e. the number of samples between two consecutive frames, is less than the frame length. Hence, the number of frames for a fault Xn is Nn = 1 + (Tn − L)/S, where · is the ﬂooring function. The frames F (matrices) are conveniently organized as vectors of dimension ˆ resized to create Z = [z1 . . . zN ] of dimension K × N . K = QL, and Z Concatenated raw front end It should be noticed that, if S = L (no overlap) and a frame is a concatenation of samples F = [xt−0.5(L−1) , . . . , xt−1 , xt , xt+1 , . . . , xt+0.5(L−1) ], ˆ would coincide, i.e. X = Z. ˆ the matrices X and Z For example, in [13] the frames are composed by the concatenation of Q = 6 raw samples and vectors z have dimension K = 198. In more details, for example, if Q = 6 (currents and voltages), a concatenated raw front end could obtain frames F of dimension 6 × 5 by concatenating to each central sample its four neighbors, two at the left and two at the right. In this case, assuming a fault with T = 10 samples and S = L = 5, one would have K = 30 and N = 2, such ˆ In this case, and Z ˆ and Z would have dimensions 6 × 10 and 30 × 2, that X = Z. respectively. Figure 6.2 illustrates the segmentation in vectors z of features for two faults. The AB fault has 2 frames while the ABC has 1. In this example, L = 2 and this leads to three vectors z, each of dimension K = 12. Framed wavelet front end When adopting multi-resolution front ends, such as wavelets, special care needs to be exercised to fully describe the processing, given their large number of degrees of freedom. It is assumed a γ-level dyadic wavelet decomposition, which has γ stages of ﬁltering and decimation [20] and transforms each of the Q waveforms into γ + 1 waveforms. More speciﬁcally, the q-th waveform is decomposed into approximation aq and details dq1 , dq2 , . . . , dqγ , for q = 1, . . . , Q. For simplicity, the dependence on q is omitted hereafter. Some works in the literature use only one of the details or calculate the average power of the coeﬃcients [21]. In contrast, the framed wavelet front end keeps all the coeﬃcients by taking in account that for γ > 1 they have diﬀerent sampling frequencies and organizing them as matrix Z. For that, instead of using a single L, the user speciﬁes a value Lmin for the waveforms with lowest fs (a and dγ ) and a larger value Li = 2γ−i Lmin is automatically adopted for the other details i = 1, . . . , γ − 1. For example, assuming γ = 3, the frame lengths for d1 and

112

Y. Pires et al.

Fig. 6.2. Organizing feature vectors z in a concatenated raw front end. In this case, the two faults with a total of three frames and L = 2 lead to three vectors z of dimension K = 12.

d2 are 4Lmin and 2Lmin , respectively. A similar reasoning is applied to the shift Si = 2γ−i Lmin , where Smin is another user-deﬁned parameter. The values are organized in a frame F of dimension Q × L, where L = 2γ Lmin . The number of frames for this organization of a wavelet decomposition is N = 1 + (Ta − Lmin )/Smin , where Ta is the number of elements in a. The notation is ﬂexible enough to easily describe several wavelet front ends, such as the concatenated wavelet (wavelet-concat, for which Lmin = Smin ) and wavelet-energy described in [22]. 6.2.2

On-Line and Post-fault Classification

Fault classiﬁcation systems can be divided into two types. The ﬁrst one aims at performing a decision (classiﬁcation) for each feature vector z or, equivalently, a frame F (giving that z is just a representation matrix F as a vector). This is typically the goal in on-line scenarios, at the level of, e.g., a protection relay [13]. Alternatively, the decision can be made at a supervisory center in a post-fault

6 Data Mining Applied to the Electric Power Industry

113

stage. The latter case makes a decision having available the whole matrix Z of variable dimension K × Nn , where n distinguishes the individual faults, which having distinct durations in the general case. The on-line and post-fault systems try to solve problems that can be cast as conventional classification [9] and sequence classification [23] problems, respectively. On-line fault classiﬁcation must be performed on a very short time span with the frame located in the beginning of the fault. It is often based on a frame corresponding to half or one cycle of the sinusoidal signal (typically of 60 or 50 Hz). For example, assuming 60 Hz and a sampling frequency of fs = 2 kHz, one cycle corresponds to L = 2000/60 ≈ 33 samples. As mentioned, on-line classiﬁcation corresponds to the conventional scenario, where one is given a training set {(z1 , y1 ), ..., (zM , yM )} containing M examples. Each example (z, y) consists of a vector z ∈ RK called instance and a label y ∈ {1, . . . , Y }. A conventional classifier is a mapping F : RK → {1 . . . , Y }. Some classiﬁers are able to provide confidence-valued scores fi (z) for each class i = 1, . . . , Y , such as a probability distribution over y. For convenience, it is assumed that all classiﬁers return a vector y with Y elements. If the classiﬁer does not naturally return conﬁdence-valued scores, the vector y is created with a unitary score for the correct class fy (z) = 1 while the others are zero fi (z) = 0, i = y. With this assumption, the ﬁnal decision is given by the max-wins rule F (z) = arg max fi (z). i

Contrasting to the on-line case, a post-fault module has to classify a sequence Z. The classiﬁer is then a mapping G : RK×N → {1, . . . ,Y } and the training set {(Z1 , y1 ), . . . ,(ZM , yM )} contains M sequences and their labels. There are techniques such as (HMM) [24] and (DTW) [25] that directly deal with sequences. Another alternative is the frame-based sequence classification (FBSC) [26], which is adopted in this work. In FBSC systems, the fault module repeatedly invokes a conventional classiﬁer F (z) (e.g., a neural network or decision tree) to obtain the scores y = (f1 (z), . . . , fY (z)) for each class. To come up with the ﬁnal decision, the fault module can then take in account the scores of all frames. Two possible options consist in calculating an accumulated score gi (Z) for each class and then using the max-wins rule G(Z) = arg max gi (Z), i

where: gi (Z) =

N

fi (zn )

(6.1)

log(fi (zn )).

(6.2)

n=1

or gi (Z) =

N n=1

As typically done, any tie when searching for the maximum can be arbitrarily broken.

114

Y. Pires et al.

A third option for combining the classiﬁers is voting, which consists in choosing the class that wins for the majority of the N frames. Voting can be mathematically described by gi (Z) =

N n=1

I(fi (zn ) == max fj (zn )), j

(6.3)

and the using the max-wins rule, where == checks for equality and I is the indicator function, which is one if its argument is true and zero otherwise. The accuracy of the system G(Z) can be evaluated according to the misclassiﬁcation rate and it is clearly dependent on the accuracy of the classiﬁer F (z). The misclassiﬁcation rates are Es and Ef , for the post-fault (sequence) and online (frame) modules, respectively. In the case of post-fault systems, in spite of Es being the actual ﬁgure of merit, it is sometimes useful to also calculate Ef . One can see that there are many degrees of freedom when designing an algorithm for fault classiﬁcation. Therefore, they must be evaluated using a comprehensive dataset and rigorous methodology. The next section presents a dataset that can be used to assess fault classiﬁcation algorithms and the experimental setup used for the experiments in this paper.

6.3 UFPAFaults: A Free Dataset of Faults in Transmission Lines UFPAFaults is a public and comprehensive labeled dataset with faults, which allows to compare algorithms and reproduce results over diﬀerent sites. It was generated by the AmazonTP software [15], which works by repeatedly invoking the ATP simulator to automatically create a large number of distinct situation. UFPAFaults is currently in its fourth version, called UFPAFaults4, and is available at www.laps.ufpa.br/freedatasets/UfpaFaults/. The UFPAFaults4 dataset is composed by 5,500 faults, organized into ﬁve sets of 100, 200, . . ., 1000 faults each. The division into these sets is to facilitate obtaining sample complexity curves, where one observes how the performance varies with the number of training examples. Each fault correspond to three voltage and three current waveforms stored as binary ﬁles with an associated text (ASCII) ﬁles, which stores a description of the fault (its endpoints, label, etc.). The waveform samples are stored as real numbers represented as the primitive type ﬂoat in Java (big-endian, 32-bits, IEEE-754 numbers). A brief explanation about how the faults are generated with AmazonTP is given in the next paragraphs. For more details, see [15]. Given an ATP ﬁle representing a electric circuit with transmission lines, AmazonTP automatically identiﬁes all occurrences of them (the current version supports only Z-T transmission lines), replaces each of them by the block described in Figure 6.3, and creates a new ATP ﬁle. Based on this master ﬁle, AmazonTP can then simulate short-circuits by closing and opening switches. For example,

6 Data Mining Applied to the Electric Power Industry

115

Fig. 6.3. Block for simulating a fault. The SW elements are switches and R are resistances.

closing switches SW-A and SW-B while leaving all others starts a AB fault. Note that each original line is replaced by two lines. The fault location is established by the lengths of these two lines. All the parameters for each fault simulation were randomly generated. The values of all four resistances were obtained as independent and identically distributed (i.i.d.) samples drawn from a uniform probability density function (pdf) U(0.1, 10), where the notation U(a, b) indicates a support of [a, b]. In other words, the resistances varied uniformly from 0.1 to 10 Ohms. The begin (from which the incidence angle depends [4]) and duration (both in seconds) of the fault were drawn from U(0.1, 0.9) and U(0.07, 0.5), respectively. The electric circuit used to create UFPAFaults4 has three Z-T lines [15]. It was assumed that the fault could occur with the same probability at any of these three lines. The fault location was drawn from U(2, 98) (percentage of the total line length). For example, assuming the faulty line has 4 km and the location is 25%, i.e., a fault at 1 km from the reference (left) side is simulated by setting the two respective Z-T lines to have 1 km and 3 km (their total length must always coincide with the length of the original line). AmazonTP generated 11 types of faults (AG, BG, CG, AB, AC, BC, ABC, ABG, ACG, BCG, ABCG) uniformly distributed. Note that ABC and ABCG are considered the same class in this work, which then has twice the number of examples of the other classes. The voltage and current waveforms generated by the ATP simulations had a sampling period equal to 0.25 microseconds, corresponding to a sampling frequency fs = 40 kHz. It is possible to obtain versions with smaller values for fs by decimating the original waveforms. This operation requires low-pass ﬁltering to avoid aliasing. Details about decimation and ﬁltering can be found in digital signal processing textbooks (e.g., [27]).

116

Y. Pires et al.

6.4 Experimental Results The experiments used the UFPAFaults4 dataset. Normalization and a rigorous model selection procedure were adopted. This section also describes the front ends, classiﬁers and their associated results. 6.4.1

Normalization

The elements of the feature vectors z may have very diﬀerent dynamic ranges (e.g., voltage in kV and currents in Amperes). This can cause the learning algorithms to perform poorly. Therefore, as a pre-processing stage, it is important to apply a normalization process. For example, the features may be normalized to the range [−1, 1], which is called per unit or pu in power systems. Besides, normalization is useful when the same algorithm should work with diﬀerent circuits, eventually with diﬀerent nominal values for voltages and currents. There are many possible algorithms for normalization of time series. This work adopted the so-called allfault [26], which takes in account all duration of the waveforms for getting the maximum and minimum amplitudes of each phase, and the converting to pu. A distinct normalization factor is calculated for each of the Q waveforms. 6.4.2

Model Selection for the FBSC Classifier

Often, the best performance of a learning algorithm on a particular dataset can only be achieved by tedious parameter tuning. This task is called model selection and corresponds, for example, to choosing parameters such as the number of neurons in the hidden layer for a neural network. A popular strategy for model selection is cross-validation [9]. This is a computationally intensive approach, but avoids tuning the parameters by repeatedly evaluating the classiﬁer using the test set. The test set should be used only once, after model selection, such that the error rate on this test set is a good indicator of the generalization capability of the learning algorithm. When dealing with frames extracted from sequences, it should be noted that, in conventional classiﬁcation, the examples are assumed to be i.i.d. “samples” from an unknown but ﬁxed distribution P (z, y). Because examples are independent, they can be arbitrarily split into training and test sets. Similarly, when organizing the folds for cross-validation, examples can be arbitrarily assigned to the training and validation fold. However, the i.i.d. assumption becomes invalid, for example, when examples (z,y) are extracted from contiguous frames of the same sequence given the relatively high similarity among them. Hence, in practice it is important to use cross-validation properly, to avoid overﬁtting due to a training set with similar vectors extracted from the same waveform. An example helps clarifying the issue. Assume a training set with 4 faults, each with 5 frames. Adopting the FBSC architecture, one could think of using all 20 examples for doing 10-folds cross-validation for model selection of a classiﬁer. In this case, each test fold would have 2 examples, potentially very similar to

6 Data Mining Applied to the Electric Power Industry

117

other examples used for training. The estimated test errors would be overly optimistic and the model selection could be far from optimal. To diminish the susceptibility to overﬁtting, the model selection could be based on a 4-folds crossvalidation with test folds composed by all 5 examples from each fault. Typically, the existing routines for cross-validation do not impose such constraint. When using simulated data, because it is easy to generate a large number of faults, it is possible to avoid the matter by randomly picking only one frame of each fault. For real (and typically scarce) data, discarding data is not feasible but one should adopt routines that do not use frames from the same fault simultaneously for training and test. This work performed model selection via a validation set, disjoint to both training and test sets. A grid (Cartesian product) of model parameters is created and the point (set of parameters) that leads to the smallest error in the validation set is selected. For each coordinate, the user speciﬁes the minimum and maximum values, the number of values and chooses between a linear or logarithmic spacing for the values. 6.4.3

Used Front Ends

For all simulations, the 40 kHz signal was low-pass ﬁltered and decimated by 20 to create a signal with fs = 2 kHz. Three diﬀerent front ends were evaluated. Their conﬁguration is summarized in Table 6.1. The ﬁrst is a concatenated raw front end with S = L. The frame length L was varied from 1, 5, 7, 9, 11 and 33. The two wavelet front ends wavelet used the Daubechies 4 [20] with a 3-level decomposition (γ = 3). Hence, for each of the Q = 6 waveforms, the wavelet decomposition generated four signals. The wavelet-energy used Lmin = Smin = 1 while wavelet-concat was tested with the conﬁgurations shown in Table 6.1. 6.4.4

Adopted Learning Algorithms

This work investigates classiﬁers for the FBSC architecture. The used classiﬁers were a subset of the ones in Weka [28]: J4.8 decision tree (an adapted version of Quinlan’s C4.5 [29]), multilayer artiﬁcial neural network (ANN) trained withbackpropagation, naive Bayes and K-nearest neighbor (KNN) [9]. The choice of these classiﬁers was based in the fact that they are popular representatives of diﬀerent learning paradigms (probabilistic, lazy, etc.). These classiﬁers were also used for evaluating on-line fault classiﬁcation, where the decisions are made on a frame-by-frame basis. The parameters obtained by model selection for each classiﬁer are summarized in Table 6.1. The KNN used the squared-error as distance measure and K = 1. The naive Bayes used Gaussian pdfs and does not have parameters to be tuned. For the ANN, H is the number of neurons in the hidden layer, N maximum the number of epochs, L the learning rate and M the momentum [9]. For J4.8, C is the conﬁdence and M the minimum number of examples in a leaf [9].

118

Y. Pires et al. Table 6.1. Summary of parameters for the front ends and two classiﬁers Front end L or Lmin 1 5 7 Raw 9 11 33 4 Wavelet- 5 7 concat 9 Wavelet- 1 energy

6.4.5

S or Smin 1 5 7 9 11 33 2 2 3 4 1

K 6 30 42 54 66 198 192 240 336 432 24

ANN -H 8 -N 1500 -L 0.2 -M 0.3 -H 20 -N 1500 -L 0.2 -M 0.3 -H 26 -N 1500 -L 0.2 -M 0.3 -H 32 -N 1500 -L 0.2 -M 0.3 -H 38 -N 1500 -L 0.2 -M 0.3 -H 104 -N 1500 -L 0.2 -M 0.3 -H 100 -N 1500 -L 0.2 -M 0.2 -H 125 -N 1500 -L 0.2 -M 0.3 -H 173 -N 1500 -L 0.2 -M 0.3 -H 248 -N 1500 -L 0.5 -M 0.4 -H 17 -N 1500 -L 0.2 -M 0.3

J4.8 -C 0.35 -M 10 -C 0.5467 -M 10 -C 0.7433 -M 10 -C 0.35 -M 10 -C 0.5467 -M 10 -C 0.35 -M 10 -C 0.5467 -M 10 -C 0.35 -M 10 -C 0.54 -M 10 -C 0.54 -M 10 -C 0.5467 -M 10

Post-fault Results

FBSC post-fault classiﬁers G were designed using the front ends and classiﬁers described in Table 6.1. Also, it was adopted the max-wins rule and Eq. 6.1. Figure 6.4 was obtained using the Ef misclassiﬁcation rate obtained with a concatenated raw front end and an ANN classiﬁer with L = S = 9. Model selection was used for each value of M , given that the best parameters for the classiﬁer typically depend on the number of training examples [30]. It can be seen that approximately M = 700 examples suﬃces to train the ANN classiﬁer. The results for frame-based classiﬁcation using the concatenated raw front end are shown in Figure 6.5. The best results were obtained by the ANN, followed by the J4.8 classiﬁer. The best frame length was L = 9. It is interesting to note that for L = 1, ANN achieved an error rate more than three times the one achiaved by J4.8.

10

10

9

9 8

6

7 Mean of Error(%)

7

Error(%)

8

Set 1 Set 2 Set 3 Set 4 Set 5

5 4

6 5 4

3

3

2

2

1

1

0 100

200

300

400

500 600 Samples

700

800

900

1000

0

100

200

300

400

500 600 Samples

700

800

900

1000

Fig. 6.4. Sample complexity for frame-based classiﬁcation (the error is Ef ) using ANN. The left-most ﬁgure shows the individual results for each of the 5 disjoint datasets and the right-most shows the average and standard deviation. It can be seen that approximately M = 700 examples suﬃces to train this classiﬁer.

6 Data Mining Applied to the Electric Power Industry

119

Fig. 6.5. Error rate Ef for several classiﬁers and frame lengths (L) using the concatenated raw front end and allfault normalization

Fig. 6.6. Error Es for post-fault classiﬁcation. The ANN-based FBSC achieved Es = 0 for L > 1

Figure 6.6 shows the actual results for post-fault (sequence) classiﬁcation. The ANN-based FBSC achieved zero errors for all values of L but L = 1. The J4.8 classiﬁer achieved an error of Es = 0.1% with a small computational cost. For J4.8, it was more accurate to use a small value of L. Figure 6.7 emphasizes the relationship between Ef and Es in Figures 6.5 and 6.6, respectively. It can be seen that, as expected, Ef is larger than Es , but the actual value depends on the classiﬁer. Figure 6.8 shows the results for two wavelets front ends. These results did not outperform the best result obtained with a concatenated raw front end. One should notice, however, that there are many degrees of freedom when designing a front end based on the wavelet transform, and the numbers in Figure 6.8 should be seen as baseline results.

120

Y. Pires et al.

Fig. 6.7. Diﬀerence Ef − Es between the error rates for frame-by-frame and sequence classiﬁcation 10 9.1

9

Lmin=5 Smin=2 Lmin=7 Smin=3

8

Lmin=9 Smin=4 Best Raw waveletenergy

7

8 7.4 6.8

7 6.5

Error(%)

6 5 4 3 2 0.7

1 0.18

0

0.11

ANN

0.09

J4.8

Fig. 6.8. Results (Ef ) for two wavelet front ends using the ANN and J4.8 classiﬁers. The best result with the concatenated raw front end is also shown, for the sake of comparison.

6.5 Conclusions This work presented a thorough description of the issues related to the design of fault classiﬁcation modules for power electric systems. The solutions to this problem involve digital signal processing and machine learning algorithms.

6 Data Mining Applied to the Electric Power Industry

121

Studies in classiﬁcation are part of a more general set of data mining tasks applied to time series representing PQ events. Current investigations also include clustering and association rules. In all these cases, the assessment of the algorithms is problematic due to the lack of labeled data. The simulations with ATP help to circumvent this problem. The release of the UFPAfaults4 dataset aims to help researchers interested in the problem. This work has established baseline results using UFPAfaults4, but some of them are not conclusive. Several issues require further investigation. The front ends, especially the one based on wavelets, require ﬁner tuning. Also, the results should indicate conﬁdence intervals based on statistical tests [9].

References 1. Bell, S., McArthur, S., McDonald, J., et al.: Model-based analysis of protection system performance. IEE Proc. Gener. Transm. Distrib. 145, 547–552 (1998) 2. Zhang, L., Chen, S., Hu, Q.: Dynamic shape modeling of consumers’ daily load based on data mining. In: Advanced Data Mining and Applications (ADMA), pp. 712–719 (2005) 3. Kitayama, M., Matsubara, R., Izui, Y.: Application of data mining to customer proﬁle analysis in the power electric industry. Power Engineering Society Winter Meeting 1, 632–634 (2002) 4. Bollen, M.H.J.: Understanding power quality problems: Voltage sags and interruptions. In: IEEE Press Series on Power Engineering (2000) 5. Santoso, S., Power Lamoree, J.D.: Power quality data analysis: From raw data to knowledge using knowledge discovery approach. In: Engineering Society Summer Meeting, pp. 172–177. IEEE, Los Alamitos (2000) 6. Vasilic, S.: Fuzzy Neural Network Pattern Recognition Algorithm for Classiﬁcation of the Events in Power System Networks. PhD thesis, Texas A&M University (2004) 7. Mitchell, T.: Machine Learning. McGraw-Hill, New York (1997) 8. Hastie, T., Tibshirani, R., Friedman, J.: The elements of statistical learning. Springer, Heidelberg (2001) 9. Witten, I., Frank, E.: Data Mining: Practical Machine Learning Tools and Techniques with Java Implementations, 2nd edn. Morgan Kaufmann, San Francisco (2005) 10. Luo, X., Kezunovic, M.: Fault analysis based on integration of digital relay and DFR. Power Engineering Society General Meeting 1, 746–751 (2005) 11. Clarke, G., Reynders, D.: Practical Modern SCADA Protocols: DNP3, 60870.5 and Related Systems. Newnes (2004) 12. Boyer, S.A.: SCADA: Supervisory Control and Data Acquisition. In: ISA-The Instrumentation, Systems, and Automation Society (1999) 13. Zhang, N., Kezunovic, M.: A real time fault analysis tool for monitoring operation of transmission line protective relay. Electric Power Systems Research 77, 361–370 (2007) 14. EMTP. Alternative Transients Program (ATP) Rule Book. Canadian/American EMTP User’s Group (1995) 15. Pires, Y., Santos, A., Borges, J., Carvalho, A., Vinicius, M., Nunes, A., Santoso, S., Klautau, A.: A framework for evaluating data mining techniques applied to power quality. In: Brazilian Conference on Neural Networks, CBRN (2005)

122

Y. Pires et al.

16. Keogh, E., Kasetty, S.: On the need for time series data mining benchmarks: A survey and empirical demonstration. In: The 8th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining., pp. 102–111 (2002) 17. McEachern, A.: A ﬂoating-window algorithm for detecting certain power line faults that disrupt sensitive electronic loads. Instrumentation and Measurement, IEEE Trans. 39, 112–115 (1990) 18. Barnett, V., Lewis, T.: Outliers in Statistical Data, 3rd edn. Wiley, Chichester (1994) 19. Anderson, P.M.: Analysis of faulted Power Systems. IEEE Press, Los Alamitos (1995) 20. Vetterli, M., Kovačević, J.: Wavelets and Subband Coding. Prentice Hall, Englewood Cliﬀs (1995) 21. Aguilera, C., Orduna, E., Rattá, G.: Fault detection, classiﬁcation and faulted phase selection approach based on high-frequency voltage signals applied to a series-compensated line. IEEE Proc.- Gener. Transm. Distrib. 153, 469–475 (2006) 22. Morais, J., Pires, Y., Cardoso, C., Klautau, A.: An experimental evaluation of automatic classiﬁcation of sequences representing short circuits in transmission lines. In: Brazilian Conference on Neural Networks, CBRN (2007) 23. Li, M., Sleep, R.: A robust approach to sequence classiﬁcation. In: International Conference on Tools with Artiﬁcial Intelligence, p. 5 (2005) 24. Rabiner, L.: A tutorial on hidden Markov models and selected applications in speech recognition. Proceedings of the IEEE 77(2), 257–286 (1989) 25. Sakoe, H., Chiba, S.: Dynamic programming algorithm optimization for spoken word recognition. IEEE Trans. on ASSP 26(1), 43–49 (1978) 26. Morais, J., Pires, Y., Cardoso, C., Klautau, A.: Data mining applied to the electric power industry: Classiﬁcation of short-circuit in transmission lines. In: IEEE International Conference on Intelligent Systems Design and Applications, ISDA (2007) 27. Oppenheim, A., Schafer, R.: Discrete-time Signal Processing. Prentice-Hall, Englewood Cliﬀs (1989) 28. http://www.cs.waikato.ac.nz/ml/weka 29. Quinlan, J.: C4.5: Programs for Machine Learning. Morgan Kaufmann, San Francisco (1993) 30. Rifkin, R., Klautau, A.: In defense of one-vs-all classiﬁcation. Journal of Machine Learning Research 5, 101–141 (2004)

Author Index

Azevedo, Denise N. Rotondi Bhatnagar, Vasudha

1

Klautau, Aldebaro Kochhar, Sarabjeet

107 83

83 Ladeira, Marcelo 17 Lechevallier, Yves 71

Cardoso, Claudomir 107 Carvalho, Rommel N. 17 Costa, Paulo Cesar G. 17 da Silva, Alzennyr 71 de Carvalho, Francisco 71 de Oliveira, Jos´e M. Parente

Matsumoto, Shou Morais, Jeﬀerson

17 107

Pires, Yomara Pozo, Aurora

107 47

Goldbarg, Elizabeth 47 Goldbarg, Marco 47

Rossi, Fabrice

71

Ishida, Celso Y.

Santos, La´ecio L.

47

1

17