Lecture Notes in Computer Science Edited by G. Goos, J. Hartmanis, and J. van Leeuwen
2534
3
Berlin Heidelberg New York Barcelona Hong Kong London Milan Paris Tokyo
Steffen Lange Ken Satoh Carl H. Smith (Eds.)
Discovery Science 5th International Conference, DS 2002 L¨ubeck, Germany, November 24-26, 2002 Proceedings
13
Series Editors Gerhard Goos, Karlsruhe University, Germany Juris Hartmanis, Cornell University, NY, USA Jan van Leeuwen, Utrecht University, The Netherlands Volume Editors Steffen Lange Deutsches Forschungszentrum f¨ur K¨unstliche Intelligenz Stuhlsatzenhausweg 3, 66123 Saarbr¨ucken, Germany E-mail:
[email protected] Ken Satoh National Institute of Informatics 2-1-2 Hitotsubashi, Chiyoda-ku, Tokyo, 101-8430, Japan E-mail:
[email protected] Carl H. Smith University of Maryland, Department of Computer Science College Park, Maryland, MD 20742, USA E-mail:
[email protected]
Cataloging-in-Publication Data applied for A catalog record for this book is available from the Library of Congress. Bibliographic information published by Die Deutsche Bibliothek Die Deutsche Bibliothek lists this publication in the Deutsche Nationalbibliografie; detailed bibliographic data is available in the Internet at
.
CR Subject Classification (1998): H.2.8, I.2, H.3, J.1, J.2 ISSN 0302-9743 ISBN 3-540-00188-3 Springer-Verlag Berlin Heidelberg New York This work is subject to copyright. All rights are reserved, whether the whole or part of the material is concerned, specifically the rights of translation, reprinting, re-use of illustrations, recitation, broadcasting, reproduction on microfilms or in any other way, and storage in data banks. Duplication of this publication or parts thereof is permitted only under the provisions of the German Copyright Law of September 9, 1965, in its current version, and permission for use must always be obtained from Springer-Verlag. Violations are liable for prosecution under the German Copyright Law. Springer-Verlag Berlin Heidelberg New York a member of BertelsmannSpringer Science+Business Media GmbH http://www.springer.de © Springer-Verlag Berlin Heidelberg 2002 Printed in Germany Typesetting: Camera-ready by author, data conversion by PTP-Berlin, Stefan Sossna e.K. Printed on acid-free paper SPIN: 10871526 06/3142 543210
Preface
This volume contains the papers presented at the 5th International Conference on Discovery Science (DS 2002) held at the M¨ ovenpick Hotel, L¨ ubeck, Germany, November 24–26, 2002. The conference was supported by CorpoBase, DFKI GmbH, and JessenLenz. The conference was collocated with the 13th International Conference on Algorithmic Learning Theory (ALT 2002). Both conferences were held in parallel and shared five invited talks as well as all social events. The combination of ALT 2002 and DS 2002 allowed for a comprehensive treatment of recent developments in computational learning theory and machine learning – some of the cornerstones of discovery science. In response to the call for papers 76 submissions were received. The program committee selected 17 submissions as regular papers and 29 submissions as poster presentations of which 27 have been submitted for publication. This selection was based on clarity, significance, and originality, as well as on relevance to the rapidly evolving field of discovery science. The conference provided an open forum for intensive discussions and interchange of new information among researchers working in the new area of discovery science. The conference focused on the following areas related to discovery: logic for/of knowledge discovery; knowledge discovery by inferences, learning algorithms, and heuristic search; scientific discovery; knowledge discovery in databases; data mining; knowledge discovery in network environments; active mining; inductive logic programming; abductive reasoning; machine learning; constructive programming as discovery; intelligent network agents; knowledge discovery from texts and from unstructured and multimedia data; statistical methods and neural networks for knowledge discovery; data and knowledge visualization; knowledge discovery and human interaction; human factors in knowledge discovery; philosophy and psychology of discovery; chance discovery; and application of knowledge discovery to natural science and social science. The proceedings contain papers from a variety of the above areas, reflecting both the theoretical and the practical aspects of discovery science. This year’s conference was the fifth in a series of annual conferences established in 1998. Continuation of this process is supervised by the DS steering committee consisting of Setsuo Arikawa (Chair, Kyushu Univ., Japan), Klaus P. Jantke (DFKI GmbH, Germany), Masahiko Sato (Kyoto Univ., Japan), Ayumi Shinohara (Kyushu Univ., Japan), Carl H. Smith (Univ. Maryland, USA), and Thomas Zeugmann (Univ. L¨ ubeck, Germany). This volume consists of three parts. The first part contains the invited talks of ALT 2002 and DS 2002. The invited talks were given by Susumu Hayashi (Kobe Univ., Japan), Rudolf Kruse (Tech. Univ. Magdeburg, Germany), John Shawe-Taylor (Royal Holloway, Univ. London, UK), Gerhard Widmer (Austrian Research Inst. for AI, Austria), and Jan Witten (Univ. Waikato, New Zealand).
VI
Preface
Since the invited talks were for both conferences, this volume contains the full versions of Rudolf Kruse’s and Gerhard Widmer’s talk as well as the abstracts of the others. The second part contains the accepted regular papers and the third part contains the written versions of the posters accepted for presentation during the conference. We would like to thank all individuals and institutions who contributed to the success of the conference: the authors of submitted papers, the invited speakers, the sponsors, and Springer-Verlag. We are particularly grateful to the members of the program committee for spending their valuable time reviewing and evaluating the submissions and for participating in online discussions, ensuring that the presentations at the conference were of high technical quality. We are also grateful to the external additional referees for their considerable contribution to this process. Last, but not least, we would like to express our immense gratitude to Andreas Jacoby (Univ. L¨ ubeck, Germany) and Thomas Zeugmann (Univ. L¨ ubeck, Germany) who did a remarkable job as local arrangement chairs for both conferences.
November 2002
Steffen Lange Ken Satoh Carl H. Smith
Organization
Conference Chair Carl H. Smith
University of Maryland, USA
Program Committee Steffen Lange (Co-chair) Ken Satoh (Co-chair) Diane J. Cook Andreas Dengel Peter A. Flach Gunter Grieser Achim Hoffmann Klaus P. Jantke John R. Josephson Pat Langley Bing Liu Heikki Mannila Hiroshi Motoda Stephan Muggleton Ryohei Nakano Yukio Ohsawa Jorge C.G. Ramirez Ayumi Shinohara Stefan Wrobel Kenji Yamanishi
DFKI GmbH, Germany Nat. Institute of Informatics, Japan Univ. of Texas at Arlington, USA DFKI GmbH, Germany Univ. of Bristol, UK TU Darmstadt, Germany UNSW, Australia DFKI GmbH, Germany Ohio State Univ., USA ISLE, USA National Univ., Singapore Helsinki Univ. of Tech., Finland Osaka Univ., Japan Imperial College, UK Nagoya Inst. Tech., Japan Tsukuba Univ., Japan ACS State Healthcare, USA Kyushu Univ., Japan Tech. Univ. Magdeburg, Germany NEC Co. Ltd., Japan
Additional Referees Hiroki Arimura Hideki Asoh Stephan Baumann Michael Boronowsky Nigel Collier Ludger van Elst Jim Farrand Koichi Furukawa Artur Garcez Elias Gyftodimos J¨ org Herrmann
Michiel de Hoon Tamas Horvath Hitoshi Iba Ryutaro Ichise Naresh Iyer Markus Junker Thomas Kieninger J¨ org Kindermann Asanobu Kitamoto Stefan Klink Willi Kl¨ osgen
VIII
Organization
Mark-A. Krogel Tim Langford Yuko Matsunaga Martin Memmel Satoru Miyano Satoshi Morinaga Yoichi Motomura Heinz M¨ uhlenbein Jochen Nessel Kouzou Ohara Gerhard Paaß Yonghong Peng
Son Bao Pham Kazumi Saito Hiroko Satoh Shinichi Shimozono Jun-ichi Takeuchi Osamu Watanabe Shaomin Wu Seiji Yamada Masayuki Yamamura Thomas Zeugmann Sandra Zilles
Local Arrangements Andreas Jacoby Thomas Zeugmann
Univ. L¨ ubeck, Germany Univ. L¨ ubeck, Germany
Table of Contents
Invited Talks Mathematics Based on Learning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Susumu Hayashi
1
Data Mining with Graphical Models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Rudolf Kruse, Christian Borgelt
2
On the Eigenspectrum of the Gram Matrix and Its Relationship to the Operator Eigenspectrum . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 John Shawe-Taylor, Chris Williams, Nello Cristianini, Jaz Kandola In Search of the Horowitz Factor: Interim Report on a Musical Discovery Project . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 Gerhard Widmer Learning Structure from Sequences, with Applications in a Digital Library . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33 Ian H. Witten
Regular Papers Application of Discovery to Natural Science Discovering Frequent Structured Patterns from String Databases: An Application to Biological Sequences . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34 Luigi Palopoli, Giorgio Terracina Discovery in Hydrating Plaster Using Machine Learning Methods . . . . . . . . 47 Judith E. Devaney, John G. Hagedorn Revising Qualitative Models of Gene Regulation . . . . . . . . . . . . . . . . . . . . . . . 59 Kazumi Saito, Stephen Bay, Pat Langley Knowledge Discovery from Unstructured and Semi-structured Data SEuS: Structure Extraction Using Summaries . . . . . . . . . . . . . . . . . . . . . . . . . 71 Shayan Ghazizadeh, Sudarshan S. Chawathe Discovering Best Variable-Length-Don’t-Care Patterns . . . . . . . . . . . . . . . . . 86 Shunsuke Inenaga, Hideo Bannai, Ayumi Shinohara, Masayuki Takeda, Setsuo Arikawa
X
Table of Contents
Meta-learning and Analysis of Machine Learning Algorithms A Study on the Effect of Class Distribution Using Cost-Sensitive Learning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 98 Kai Ming Ting Model Complexity and Algorithm Selection in Classification . . . . . . . . . . . . . 113 Melanie Hilario Experiments with Projection Learning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 127 Tapio Elomaa, J.T. Lindgren Improved Dataset Characterisation for Meta-learning . . . . . . . . . . . . . . . . . . . 141 Yonghong Peng, Peter A. Flach, Carlos Soares, Pavel Brazdil Combining Machine Learning Algorithms Racing Committees for Large Datasets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 153 Eibe Frank, Geoffrey Holmes, Richard Kirkby, Mark Hall From Ensemble Methods to Comprehensible Models . . . . . . . . . . . . . . . . . . . . 165 C. Ferri, J. Hern´ andez-Orallo, M.J. Ram´ırez-Quintana Neural Networks and Statistical Methods Learning the Causal Structure of Overlapping Variable Sets . . . . . . . . . . . . 178 David Danks Extraction of Logical Rules from Data by Means of Piecewise-Linear Neural Networks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 192 Martin Holeˇ na Structuring Neural Networks through Bidirectional Clustering of Weights . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 206 Kazumi Saito, Ryohei Nakano New Approaches to Knowledge Discovery Toward Drawing an Atlas of Hypothesis Classes: Approximating a Hypothesis via Another Hypothesis Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . 220 Osamu Maruyama, Takayoshi Shoudai, Satoru Miyano Datascape Survey Using the Cascade Model . . . . . . . . . . . . . . . . . . . . . . . . . . . 233 Takashi Okada Learning Hierarchical Skills from Observation . . . . . . . . . . . . . . . . . . . . . . . . . 247 Ryutaro Ichise, Daniel Shapiro, Pat Langley
Table of Contents
XI
Poster Papers Applications of Knowledge Discovery to Natural Science Image Analysis for Detecting Faulty Spots from Microarray Images . . . . . . 259 Salla Ruosaari, Jaakko Hollm´en Inferring Gene Regulatory Networks from Time-Ordered Gene Expression Data Using Differential Equations . . . . . . . . . . . . . . . . . . . . . . . . . . 267 Michiel de Hoon, Seiya Imoto, Satoru Miyano DNA-Tract Curvature Profile Reconstruction: A Fragment Flipping Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 275 Daniele Masotti Evolution Map: Modeling State Transition of Typhoon Image Sequences by Spatio-Temporal Clustering . . . . . . . . . . . . . . . . . . . . . . . 283 Asanobu Kitamoto Structure-Sweetness Relationships of Aspartame Derivatives by GUHA . . . 291 Jaroslava Halova, Premysl Zak, Pavel Stopka, Tomoaki Yuzuri, Yukino Abe, Kazuhisa Sakakibara, Hiroko Suezawa, Minoru Hirota Knowledge Discovery from Texts A Hybrid Approach for Chinese Named Entity Recognition . . . . . . . . . . . . . . 297 Xiaoshan Fang, Huanye Sheng Extraction of Word Senses from Human Factors in Knowledge Discovery . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 302 Yoo-Jin Moon, Minkoo Kim, Youngho Hwang, Pankoo Kim, Kijoon Choi Event Pattern Discovery from the Stock Market Bulletin . . . . . . . . . . . . . . . . 310 Fang Li, Huanye Sheng, Dongmo Zhang Email Categorization Using Fast Machine Learning Algorithms . . . . . . . . . . 316 Jihoon Yang, Sung-Yong Park Discovery of Maximal Analogies between Stories . . . . . . . . . . . . . . . . . . . . . . . 324 Makoto Haraguchi, Shigetora Nakano, Masaharu Yoshioka Automatic Wrapper Generation for Multilingual Web Resources . . . . . . . . . 332 Yasuhiro Yamada, Daisuke Ikeda, Sachio Hirokawa Combining Multiple K-Nearest Neighbor Classifiers for Text Classification by Reducts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 340 Yongguang Bao, Naohiro Ishii
XII
Table of Contents
ARISTA Causal Knowledge Discovery from Texts . . . . . . . . . . . . . . . . . . . . . . 348 John Kontos, Areti Elmaoglou, Ioanna Malagardi Applications of Knowledge Discovery to Social Science Knowledge Discovery as Applied to Music: Will Music Web Retrieval Revolutionize Musicology? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 356 Francis Rousseaux, Alain Bonardi Process Mining: Discovering Direct Successors in Process Logs . . . . . . . . . . . 364 Laura Maruster, A.J.M.M. Weijters, W.M.P. van der Aalst, Antal van den Bosch The Emergence of Artificial Creole by the EM Algorithm . . . . . . . . . . . . . . . 374 Makoto Nakamura, Satoshi Tojo Generalized Musical Pattern Discovery by Analogy from Local Viewpoints . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 382 Olivier Lartillot Machine Learning Approaches Using Genetic Algorithms-Based Approach for Better Decision Trees: A Computational Study . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 390 Zhiwei Fu Handling Feature Ambiguity in Knowledge Discovery from Time Series . . . 398 Frank H¨ oppner A Compositional Framework for Mining Longest Ranges . . . . . . . . . . . . . . . . 406 Haiyan Zhao, Zhenjiang Hu, Masato Takeichi Post-processing Operators for Browsing Large Sets of Association Rules . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 414 Alipio Jorge, Jo˜ ao Po¸cas, Paulo Azevedo Mining Patterns from Structured Data by Beam-Wise Graph-Based Induction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 422 Takashi Matsuda, Hiroshi Motoda, Tetsuya Yoshida, Takashi Washio Feature Selection for Propositionalization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 430 Mark-A. Krogel, Stefan Wrobel Subspace Clustering Based on Compressibility . . . . . . . . . . . . . . . . . . . . . . . . . 435 Masaki Narahashi, Einoshin Suzuki
Table of Contents
XIII
New Approaches to Knowledge Discovery The Extra-Theoretical Dimension of Discovery. Extracting Knowledge by Abduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 441 Lorenzo Magnani, Matteo Piazza, Riccardo Dossena Discovery Process on the WWW: Analysis Based on a Theory of Scientific Discovery . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 449 Hitomi Saito, Kazuhisa Miwa Invention vs. Discovery (A Critical Discussion) . . . . . . . . . . . . . . . . . . . . . . . . 457 Carlotta Piscopo, Mauro Birattari Author Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 463
Mathematics Based on Learning Susumu Hayashi Kobe University, Rokko-dai, Nada, Kobe 657-8501, Japan, [email protected], http://www.shayashi.jp
Abstract. Learning theoretic aspects of mathematics and logic have been studied by many authors. They study how mathematical and logical objects are algorithmically “learned” (inferred) from finite data. Although the subjects of studies are mathematical objects, the objective of the studies are learning. In this paper, a mathematics of which foundation itself is learning theoretic will be introduced. It is called LimitComputable Mathematics. It was originally introduced as a means for “Proof Animation,” which is expected to make interactive formal proof development easier. Although the original objective was not learning theoretic at all, learning theory is indispensable for our research.
The full version of this paper is published in the Proceedings of the 13th International Conference on Algorithmic Learning Theory, Lecture Notes in Artificial Intelligence Vol. 2533
S. Lange, K. Satoh, and C.H. Smith (Eds.): DS 2002, LNCS 2534, p. 1, 2002. c Springer-Verlag Berlin Heidelberg 2002
Data Mining with Graphical Models Rudolf Kruse and Christian Borgelt Department of Knowledge Processing and Language Engineering Otto-von-Guericke-University of Magdeburg Universit¨ atsplatz 2, D-39106 Magdeburg, Germany {kruse,borgelt}@iws.cs.uni-magdeburg.de
Abstract. Data Mining, or Knowledge Discovery in Databases, is a fairly young research area that has emerged as a reply to the flood of data we are faced with nowadays. It tries to meet the challenge to develop methods that can help human beings to discover useful patterns in their data. One of these techniques — and definitely one of the most important, because it can be used for such frequent data mining tasks like classifier construction and dependence analysis — is learning graphical models from datasets of sample cases. In this paper we review the ideas underlying graphical models, with a special emphasis on the less well known possibilistic networks. We discuss the main principles of learning graphical models from data and consider briefly some algorithms that have been proposed for this task as well as data preprocessing methods and evaluation measures.
1
Introduction
Today every company stores and processes its data electronically, in production, marketing, stock-keeping or personnel management. The data processing systems used were developed, because it is very important for a company to be able to retrieve certain pieces of information, like the address of a customer, in a fast and reliable way. Today, however, with ever increasing computer power and due to advances in database and software technology, we may think about using electronically stored data not only to retrieve specific information, but also to search for hidden patterns and regularities. If, for example, by analyzing customer receipts a supermarket chain finds out that certain products are frequently bought together, turnover may be increased by placing the products on the shelves of the supermarkets accordingly. Unfortunately, in order to discover such knowledge in databases the retrieval capacities of normal database systems as well as the methods of classical data analysis are often insufficient. With them, we may retrieve arbitrary individual information, compute simple aggregations, or test the hypothesis whether the day of the week has an influence on the product quality. But more general patterns, structures, or regularities go undetected. These patterns, however, are often highly valuable and may be exploited, for instance, to increase sales. As a consequence a new research area has emerged in recent years—often called S. Lange, K. Satoh, and C.H. Smith (Eds.): DS 2002, LNCS 2534, pp. 2–11, 2002. c Springer-Verlag Berlin Heidelberg 2002
Data Mining with Graphical Models
3
Knowledge Discovery in Databases (KDD) or Data Mining (DM)—in which hypotheses and models describing the regularities in a given dataset are generated and tested automatically. The hypotheses and models found in this way can then be used to gain insight into the domain under consideration, to predict its future development, and to support decision making. In this paper we consider two of the most important data mining tasks, namely the construction of classifiers and the analysis of dependences. Among the different methods for these tasks we concentrate on learning a graphical model from a dataset of sample cases. Furthermore, our emphasis is on possibilistic graphical models, which are a powerful tool for the analysis of imprecise data.
2
Graphical Models
An object or a case of a given domain of interest is usually described by a set of attributes. For instance, to describe a car we may use the manufacturer, the model name, the color etc. Depending on the specific object or case under consideration these attributes have certain values, for example, Volkswagen, Golf, red etc. Sometimes only certain combinations of attribute values are possible, for example, because certain special equipment items may not be chosen simultaneously, or certain combinations of attribute values are more frequent than others, for example, red VW Golf are more frequent than yellow BMW Z1. Such possibility or frequency information can be represented as a distribution on the Cartesian product of the attribute domains. That is, to each combination of attribute values we assign its possibility or probability. Often a very large number of attributes is necessary to describe a given domain of interest appropriately. Since the number of possible value combinations grows exponentially with the number of attributes, it is often impossible to represent this distribution directly, for example, in order to draw inferences. One way to cope with this problem is to construct a graphical model. Graphical models are based on the idea that independences between attributes can be exploited to decompose a high-dimensional distributions into a set of (conditional or marginal) distributions on low-dimensional subspaces. This decomposition (as well as the independences that make it possible) is encoded by a graph: Each node represents an attribute. Edges connect nodes that are directly dependent on each other. In addition, the edges specify the paths on which evidence has to be propagated if inferences are to be drawn. Since graphical models have been developed first in probability theory and statistics, the best-known approaches originated from this area, namely Bayes networks [Pearl 1988] and Markov networks [Lauritzen and Spiegelhalter 1988]. However, the underlying decomposition principle has been generalized, resulting in the so-called valuation-based networks [Shenoy 1992], and has been transferred to possibility theory [Gebhardt and Kruse 1996]. All of these approaches lead to efficient implementations, for example, HUGIN [Andersen et al. 1989], PULCINELLA [Saffiotti and Umkehrer 1991], PATHFINDER [Heckerman 1991], and POSSINFER [Gebhardt and Kruse 1996].
4
2.1
R. Kruse and C. Borgelt
Decomposition
The notion of decomposition is probably best-known from relational database theory. Thus it comes as no surprise that relational database theory is closely connected to the theory of graphical models. This connection is based on the notion of a relation being join-decomposable, which is used in relational database systems to decompose high-dimensional relations and thus to store them with less redundancy and (of course) using less storage space. Join-decomposability means that a relation can be reconstructed from certain projections by forming the so-called natural join of these projections. Formally, this can be described as follows: Let U = {A1 , . . . , An } be a set of attributes with respective domains dom(Ai ). Furthermore let rU be a relation over U . Such a relation can be described by its indicator function, which assigns a value of 1 to all tuples that are contained in the relation and a value of 0 to all other tuples. The tuples themselves are represented as conjunctions Ai ∈U Ai = ai , which state a value for each attribute. Then the projection onto a subset M ⊆ U of the attributes can then be defined as the relation rM max Ai = ai = rU Ai = ai , Ai ∈M
∀Aj ∈U −M : ai ∈dom(Aj )
Ai ∈U
where the somewhat sloppy notation under the maximum operator is meant to express that the maximum has to be taken over all values of all attributes in the set U − M . With this notation a relation rU is called join-decomposable w.r.t. a family M = {M1 , . . . , Mm } of subsets of U if and only if ∀a1 ∈ dom(A1 ) : . . . ∀an ∈ dom(An ) : rU Ai = ai = min rM Ai = ai . Ai ∈U
M ∈M
Ai ∈M
Note that the minimum of the projections is equivalent to the natural join of relational calculus, justifying the usage of the term “join-decomposable”. This decomposition scheme can easily be transferred to the probabilistic case: All we have to do is to replace the projection operation and the natural join by their probabilistic counterparts. Thus we arrive at the decomposition formula ∀a1 ∈ dom(A1 ) : . . . ∀an ∈ dom(An ) : pU Ai = ai = φM Ai = ai . Ai ∈U
M ∈M
Ai ∈M
The functions φM can be computed from the marginal distributions on the attribute sets M . This demonstrates that the computation of a marginal distribution takes the place of the projection operation. These functions are called factor potentials [Castillo et al. 1997]. Alternatively, one may describe a decomposition of a probability distribution by exploiting the (generalized) product rule of probability theory and by using conditional distributions.
Data Mining with Graphical Models
5
The possibilistic case is even closer to the relational one, because the decomposition formula is virtually identical. The only difference is that the relations r are replaced by possibility distributions π, i.e., by functions which are not restricted to the values 0 and 1 (like indicator functions), but may take arbitrary values from the interval [0, 1]. In this way a “gradual possibility” is modeled with a generalized indicator function. As a consequence possibilistic graphical models may be seen as “fuzzifications”’ of relational graphical models. Of course, if such degrees of possibility are introduced, the question of their interpretation arises, because possibility is an inherently two-valued concept. In our research we rely on the context model [Gebhardt and Kruse 1993] to answer this question. However, since the common ways of justifying the maximum and minimum operations are not convincing, we have developed a different justification that is based on the goal of reasoning with graphical models. Details about this justification can be found in [Borgelt and Kruse 2002]. 2.2
Graphical Representation
Decompositions can very conveniently be represented by graphs. In the first place, graphs can be used to specify the sets M of attributes underlying the decomposition. How this is done depends on whether the graph is directed or undirected. If it is undirected, the sets M are the maximal cliques of the graph, where a clique is a complete subgraph, which is called maximal if it is not a proper part of another complete subgraph. If the graph is directed, we can be more explicit about the distributions of the decomposition: We can employ conditional distributions, because the direction of the edges allows us to distinguish between conditioned and conditioning attributes. However, in the relational and the possibilistic case no changes result from this, since the conditional distributions are identical to their unconditional analogs (because in these calculi no renormalization is carried out). Secondly, graphs can be used to represent (conditional) dependences and independences via the notion of node separation. What is to be understood by “separation” again depends on whether the graph is directed or undirected. If it is undirected, node separation is defined as follows: If X, Y , and Z are three disjoint sets of nodes, then Z separates X and Y if all paths from a node in X to a node in Y contain a node in Z. For directed acyclic graphs node separation is defined as follows [Pearl 1988]: If X, Y , and Z are three disjoint sets of nodes, then Z separates X and Y if there is no path (disregarding the directionality of the edges) from a node in X to a node in Y along which the following two conditions hold: 1. Every node, at which the edges of the path converge, either is in Z or has a descendant in Z, and 2. every other node is not in Z. With the help of these separation criteria we can define conditional independence graphs: A graph is a conditional independence graph w.r.t. a given (multidimensional) distribution if it captures by node separation only valid conditional
6
R. Kruse and C. Borgelt
independences. Conditional independence means (for three attributes A, B, and C with A being independent of C given B; the generalization is obvious), that P (A = a, B = b, C = c) = P (A = a | B = b) · P (C = c | B = b) in the probabilistic case and π(A = a, B = b, C = c) = min{π(A = a | B = b), π(C = c | B = b)} in the possibilistic and the relational case. These formula also indicate that conditional independence and decomposability are closely connected. Formally, this connection is established by theorems, which state that a distribution is decomposable w.r.t. a given graph if the graph is a conditional independence graph. In the probabilistic case such a theorem is usually attributed to [Hammersley and Clifford 1971]. In the possibilistic case an analogous theorem can be proven, although some restrictions have to be introduced on the graphs [Gebhardt 1997, Borgelt and Kruse 2002]. Finally, the graph underlying a graphical model is very useful to derive evidence propagation algorithms, because transmitting evidence information can be implemented by node processors that communicate by sending message to each other along the edges of the graph. Details about these methods can be found, for instance, in [Castillo et al. 1997].
3
Learning Graphical Models from Data
Since a graphical model represents the dependences and independences that hold in a given domain of interest in a very clear way and allows for efficient reasoning, it is a very powerful tool—once it is constructed. However, its construction by human experts can be tedious and time-consuming. As a consequence recent research in graphical models has placed a strong emphasis on learning graphical models from a dataset of sample cases. Although it has been shown that this learning task is NP-hard in general [Chickering et al. 1994], some very successful heuristic algorithms have been developed [Cooper and Herskovits 1992, Heckerman et al. 1995, Gebhardt and Kruse 1995]. However, some of these approaches, especially probabilistic ones, are restricted to learning from precise data. That is, the description of the sample cases must contain neither missing values nor set-valued information. There must be exactly one value for each attribute in each of the sample cases. Unfortunately, this prerequisite is rarely met in applications: Real-world databases are often incomplete and useful imprecise information (sets of values for an attribute) is frequently available (even though it is often neglected, because common database systems cannot handle it adequately). Therefore we face the challenge to extend the existing learning algorithms to incomplete and imprecise data. Research in probabilistic graphical models tries to meet this challenge mainly with the expectation maximization (EM) algorithm [Dempster et al. 1977, Bauer et al. 1997]. In our own research, however, we focus on possibilistic graphical
Data Mining with Graphical Models
7
models, because possibility theory [Dubois and Prade 1988] allows for a very convenient treatment of missing values and imprecise data. For possibilistic networks no iterative procedure like the EM algorithm is necessary, so that considerable gains in efficiency can result [Borgelt and Kruse 2002]. 3.1
Learning Principles
There are basically three approaches to learn a graphical model from data: – Test whether a given distribution is decomposable w.r.t. a given graph. – Construct a conditional independence graph through conditional independence tests. – Choose edges based on a measurement of the strength of marginal dependence of attributes. Unfortunately, none of these approaches is perfect. The first approach fails, because the number of possible graphs grows over-exponentially with the number of attributes and so it is impossible to inspect all of these graphs. The second approach usually starts from the strong assumption that the conditional independences can be represented perfectly and may require independence tests of high order, which are sufficiently reliable only if the datasets are very large. Examples in which the third approach yields a suboptimal result can easily be found [Borgelt and Kruse 2002]. Nevertheless, the second and the third approach, enhanced by additional assumptions, lead to good heuristic algorithms, which usually consists of two ingredients: 1. an evaluation measure (to assess the quality of a given model) and 2. a search method (to traverse the space of possible models). This characterization is apt, even though not all algorithms search the space of possible graphs directly. For instance, some search for conditional independences and some for the best set of parents for a given attribute. Nevertheless, all employ some search method and an evaluation measure. 3.2
Computing Projections
Apart from the ingredients of a learning algorithm for graphical models that are mentioned in the preceding section, we need an operation for a technical task, namely the estimation of the conditional or marginal distributions from a dataset of sample cases. This operation is often neglected, because it is trivial in the relational and the probabilistic case, at least for precise data. In the former it is an operation of relational calculus (namely the relational projection operations, which is why we generally call this operation a projection), in the latter it consists in counting sample cases and computing relative frequencies. Only if imprecise information is present, this operation is more complex. In this case the expectation maximization algorithm [Dempster et al. 1977, Bauer et al. 1997] is drawn upon, which can be fairly costly.
8
R. Kruse and C. Borgelt
In possibility theory the treatment of imprecise information is much simpler, especially if it is based on the context model. In this case each example case can be seen as a context, which allows to handle the imprecision conveniently inside the context. Unfortunately, computing projections in the possibilistic case is also not without problems: There is no simple operation (like simple counting), with which the marginal possibility distribution can be derived directly from the dataset to learn from. A simple example illustrating this can be found in [Borgelt and Kruse 2002]. However, we have developed a preprocessing method, which computes the closure under tuple intersection of the dataset of sample cases. From this closure the marginal distributions can be computed with a simple maximum operation in a highly efficient way [Borgelt and Kruse 2002]. 3.3
Evaluation Measures
An evaluation measure (or scoring function) serves the purpose to assess the quality of a given candidate graphical model w.r.t. a dataset of sample cases, so that it can be determined which model best fits the data. A desirable property of an evaluation measure is decomposability. That is, the quality of the model as a whole should be computable from local scores, for instance, scores for cliques or even scores for single edges. Most evaluation measures that exhibit this property measure the strength of dependence of attributes, because this is necessary for the second as well as the third approach to learning graphical models from data (cf. section 3.1), either to assess whether a conditional independence holds or to find the strongest dependences between attributes. For the probabilistic case there is a large variety of evaluation measures, which are based on a wide range of ideas and which have been developed for very different purposes. In particular all measures that have been developed for the induction of decision trees can be transferred to learning graphical models, even though this possibility is rarely fully recognized and exploited accordingly. In our research we have collected and studied several measures (e.g., information gain (ratio), Gini index, relieff measure, K2 metric and its generalization, minimum description length etc). This collection together with detailed explanations of the underlying ideas can be found in [Borgelt and Kruse 2002]. Furthermore we have developed an extension of the K2 metric [Cooper and Herskovits 1992, Heckerman et al. 1995] and an extension of measure that is based on the minimum description length principle [Rissanen 1983]. In these extensions we added a “sensitivity parameter”, which enables us to control the tendency to add further edges to the model. Such a parameter has proven highly useful in applications (cf. the application at DaimlerChrysler, briefly described in section 4). Evaluation measures for possibilistic graphical models can be derived in two ways: In the first place, the close connection to relational networks can be exploited by drawing on the notion of an α-cut, which is well known from the theory of fuzzy sets [Kruse et al. 1994]. With this notion possibility distributions can be interpreted as a set of relations, with one relation for each possibility degree α. Then it is easy to see that a possibility distribution is decomposable if and only if each of its α-cuts is decomposable. As a consequence evaluation measures for
Data Mining with Graphical Models
9
possibilistic graphical models can be derived from corresponding measures for relational graphical models by integrating over all possible values α. An example of such a measure is the specificity gain [Gebhardt 1997], which can be derived from the Hartley information gain [Hartley 1928], a measure for relational graphical models. Variants of the specificity gain, which results from different ways of normalizing it, are discussed in [Borgelt and Kruse 2002]. Another possibility to obtain evaluation measures for possibilistic networks is to form analogs of probabilistic measures. In these analogs usually a product is replaced by a minimum and a sum by a maximum. Examples of measures derived in this way can also be found in [Borgelt and Kruse 2002]. 3.4
Search Methods
The search method used determines which graphs are considered. Since an exhaustive search incurs prohibitively large costs due to the extremely high number of possible graphs, heuristic methods have to be drawn upon. These methods usually restrict the set of considered graphs considerably and use the value of the evaluation measure to guide the search. In addition, they are often greedy w.r.t. the model quality in order to speed up the search. The simplest search methods is the construction of an optimal spanning tree for given edges weights. This method was used first by [Chow and Liu 1968] with Shannon information gain providing the edge weights. In the possibilistic case the information gain may be replaced with the abovementioned specificity in order to obtain an analogous algorithm [Gebhardt 1997]. However, almost all other measures (probabilistic as well as possibilistic) are usable as well. A straightforward extension of this method is a greedy search for parent nodes in directed graphs, which often starts from a topological order of the attributes that is fixed in advance: At the beginning the evaluation measure is computed for a parentless node. Then parents are added step by step, each time selecting the attribute that yields the highest value of the evaluation measure. The search is terminated if no other parent candidates are available, a user-defined maximum number of parents is reached, or the value of the evaluation measures does not improve anymore. This search method is employed in the K2 algorithm [Cooper and Herskovits 1992] together with the K2 metric as the evaluation measure. Like optimum weight spanning tree this learning approach can easily be transferred to the possibilistic case by replacing the evaluation measure. In our research we have also developed two other search methods. The first starts from an optimal spanning tree (see above) and adds edges if conditional independences that are represented by the tree not hold. However, the edges that may be added have to satisfy certain constraints, which ensure that the cliques of the resulting graph contain at most three nodes. In addition, these constraints guarantee that the resulting graph has hypertree structure. (A hypertree is an acyclic hypergraph, and in a hypergraph the restriction that an edge connects exactly two nodes is relaxed: A hyperedge may connect an arbitrary number of nodes.) The second methods uses the well-known simulated annealing approach to learn a hypertree directly. The main problem in developing this approach
10
R. Kruse and C. Borgelt
was to find a method for randomly generating and modifying hypertrees that is sufficiently unbiased. These two search methods are highly useful, because they allow us to control the complexity of later inferences with the graphical model at learning time. The reason is that this complexity depends heavily on the size of the hyperedges of the learned hypertree, which can be easily constrained in these approaches.
4
Application
In a cooperation between the University of Magdeburg and the DaimlerChrysler corporation we had the opportunity to apply our algorithms for learning graphical models to a real-world car database. The objective of the analysis was to uncover possible causes for faults and damages. Although the chosen approach was very simple (we learned a two-layered network with one layer describing the equipment of the car and the other possible faults and damages), it was fairly successful. With a prototype implementation of several learning algorithms, we ran benchmark tests against human expert knowledge. We could easily and efficiently find hints to possible causes, which had taken human experts weeks to discover. The sensitivity parameters which we introduced into two evaluation measures (cf. section 3.3) turned out to be very important for this success.
References [Andersen et al. 1989] S.K. Andersen, K.G. Olesen, F.V. Jensen, and F. Jensen. HUGIN — A Shell for Building Bayesian Belief Universes for Expert Systems. Proc. 11th Int. J. Conf. on Artificial Intelligence (IJCAI’89, Detroit, MI, USA), 1080–1085. Morgan Kaufmann, San Mateo, CA, USA 1989 [Baldwin et al. 1995] J.F. Baldwin, T.P. Martin, and B.W. Pilsworth. FRIL — Fuzzy and Evidential Reasoning in Artificial Intelligence. Research Studies Press/J. Wiley & Sons, Taunton/Chichester, United Kingdom 1995 [Bauer et al. 1997] E. Bauer, D. Koller, and Y. Singer. Update Rules for Parameter Estimation in Bayesian Networks. Proc. 13th Conf. on Uncertainty in Artificial Intelligence (UAI’97, Providence, RI, USA), 3–13. Morgan Kaufmann, San Mateo, CA, USA 1997 [Borgelt and Kruse 2002] C. Borgelt and R. Kruse. Graphical Models — Methods for Data Analysis and Mining. J. Wiley & Sons, Chichester, United Kingdom 2002 [Castillo et al. 1997] E. Castillo, J.M. Gutierrez, and A.S. Hadi. Expert Systems and Probabilistic Network Models. Springer-Verlag, New York, NY, USA 1997 [Chickering et al. 1994] D.M. Chickering, D. Geiger, and D. Heckerman. Learning Bayesian Networks is NP-Hard (Technical Report MSR-TR-94-17). Microsoft Research, Advanced Technology Division, Redmond, WA, USA 1994 [Chow and Liu 1968] C.K. Chow and C.N. Liu. Approximating Discrete Probability Distributions with Dependence Trees. IEEE Trans. on Information Theory 14(3):462–467. IEEE Press, Piscataway, NJ, USA 1968 [Cooper and Herskovits 1992] G.F. Cooper and E. Herskovits. A Bayesian Method for the Induction of Probabilistic Networks from Data. Machine Learning 9:309–347. Kluwer, Dordrecht, Netherlands 1992
Data Mining with Graphical Models
11
[Dempster et al. 1977] A.P. Dempster, N. Laird, and D. Rubin. Maximum Likelihood from Incomplete Data via the EM Algorithm. Journal of the Royal Statistical Society (Series B) 39:1–38. Blackwell, Oxford, United Kingdom 1977 [Dubois and Prade 1988] D. Dubois and H. Prade. Possibility Theory. Plenum Press, New York, NY, USA 1988 [Dubois et al. 1996] D. Dubois, H. Prade, and R. Yager, eds. Fuzzy Set Methods in Information Engineering: A Guided Tour of Applications. J. Wiley & Sons, New York, NY, USA 1996 [Gebhardt 1997] J. Gebhardt. Learning from Data: Possibilistic Graphical Models. Habilitation Thesis, University of Braunschweig, Germany 1997 [Gebhardt and Kruse 1993] J. Gebhardt and R. Kruse. The Context Model — An Integrating View of Vagueness and Uncertainty. Int. Journal of Approximate Reasoning 9:283–314. North-Holland, Amsterdam, Netherlands 1993 [Gebhardt and Kruse 1995] J. Gebhardt and R. Kruse. Learning Possibilistic Networks from Data. Proc. 5th Int. Workshop on Artificial Intelligence and Statistics (Fort Lauderdale, FL, USA), 233–244. Springer-Verlag, New York, NY, USA 1995 [Gebhardt and Kruse 1996] J. Gebhardt and R. Kruse. POSSINFER — A Software Tool for Possibilistic Inference. In: [Dubois et al. 1996], 407–418 [Hartley 1928] R.V.L. Hartley. Transmission of Information. The Bell System Technical Journal 7:535–563. Bell Laboratories, Murray Hill, NJ, USA 1928 [Hammersley and Clifford 1971] J.M. Hammersley and P.E. Clifford. Markov Fields on Finite Graphs and Lattices. Unpublished manuscript, 1971. Cited in: [Isham 1981] [Heckerman 1991] D. Heckerman. Probabilistic Similarity Networks. MIT Press, Cambridge, MA, USA 1991 [Heckerman et al. 1995] D. Heckerman, D. Geiger, and D.M. Chickering. Learning Bayesian Networks: The Combination of Knowledge and Statistical Data. Machine Learning 20:197–243. Kluwer, Dordrecht, Netherlands 1995 [Isham 1981] V. Isham. An Introduction to Spatial Point Processes and Markov Random Fields. Int. Statistical Review 49:21–43. Int. Statistical Institute, Voorburg, Netherlands 1981 [Kruse et al. 1994] R. Kruse, J. Gebhardt, and F. Klawonn. Foundations of Fuzzy Systems, J. Wiley & Sons, Chichester, United Kingdom 1994. [Lauritzen and Spiegelhalter 1988] S.L. Lauritzen and D.J. Spiegelhalter. Local Computations with Probabilities on Graphical Structures and Their Application to Expert Systems. Journal of the Royal Statistical Society, Series B, 2(50):157–224. Blackwell, Oxford, United Kingdom 1988 [Pearl 1988] J. Pearl. Probabilistic Reasoning in Intelligent Systems: Networks of Plausible Inference. Morgan Kaufmann, San Mateo, CA, USA 1988 (2nd edition 1992) [Rissanen 1983] J. Rissanen. A Universal Prior for Integers and Estimation by Minimum Description Length. Annals of Statistics 11:416–431. Institute of Mathematical Statistics, Hayward, CA, USA 1983 [Saffiotti and Umkehrer 1991] A. Saffiotti and E. Umkehrer. PULCINELLA: A General Tool for Propagating Uncertainty in Valuation Networks. Proc. 7th Conf. on Uncertainty in Artificial Intelligence (UAI’91, Los Angeles, CA, USA), 323–331. Morgan Kaufmann, San Mateo, CA, USA 1991 [Shenoy 1992] P.P. Shenoy. Valuation-based Systems: A Framework for Managing Uncertainty in Expert Systems. In: [Zadeh and Kacprzyk 1992], 83–104 [Zadeh and Kacprzyk 1992] L.A. Zadeh and J. Kacprzyk. Fuzzy Logic for the Management of Uncertainty. J. Wiley & Sons, New York, NY, USA 1992
On the Eigenspectrum of the Gram Matrix and Its Relationship to the Operator Eigenspectrum John Shawe-Taylor1 , Chris Williams2 , Nello Cristianini3 , and Jaz Kandola1 1 Department of Computer Science, Royal Holloway, University of London Egham, Surrey TW20 0EX, UK 2 Division of Informatics, University of Edinburgh 3 Department of Statistics, University of California at Davies
Abstract. In this paper we analyze the relationships between the eigenvalues of the m × m Gram matrix K for a kernel k(·, ·) corresponding to a sample x1 , . . . , xm drawn from a density p(x) and the eigenvalues of the corresponding continuous eigenproblem. We bound the differences between the two spectra and provide a performance bound on kernel PCA.
The full version of this paper is published in the Proceedings of the 13th International Conference on Algorithmic Learning Theory, Lecture Notes in Artificial Intelligence Vol. 2533
S. Lange, K. Satoh, and C.H. Smith (Eds.): DS 2002, LNCS 2534, p. 12, 2002. c Springer-Verlag Berlin Heidelberg 2002
In Search of the Horowitz Factor: Interim Report on a Musical Discovery Project Gerhard Widmer Department of Medical Cybernetics and Artificial Intelligence, University of Vienna, Austria, and Austrian Research Institute for Artificial Intelligence, Vienna [email protected]
Abstract. The paper gives an overview of an inter-disciplinary research project whose goal is to elucidate the complex phenomenon of expressive music performance with the help of machine learning and automated discovery methods. The general research questions that guide the project are laid out, and some of the most important results achieved so far are briefly summarized (with an emphasis on the most recent and still very speculative work). A broad view of the discovery process is given, from data acquisition issues through data visualization to inductive model building and pattern discovery. It is shown that it is indeed possible for a machine to make novel and interesting discoveries even in a domain like music. The report closes with a few general lessons learned and with the identification of a number of open and challenging research problems.
1
Introduction
The potential of machine learning and automatic scientific discovery for various branches of science has been convincingly demonstrated in recent years, mainly in the natural sciences ((bio)chemistry, genetics, physics, etc. (e.g., [10,11,17,25, 29,30,31]). But is computer-based scientific discovery also possible in less easily quantifiable domains like the arts? This paper presents the latest results of a long-term interdisciplinary research project that uses AI technology to investigate one of the most fascinating — and at the same time highly elusive — phenomena in music: expressive music performance. We study how skilled musicians (concert pianists, in particular) make music ‘come alive’, how they express and communicate their understanding of the musical and emotional content of the pieces by shaping various parameters like tempo, timing, dynamics, articulation, etc. The reader will be taken on a grand tour of a complex discovery enterprise, from the intricacies of data gathering (which already require new AI methods) through novel approaches to data visualization all the way to automated data analysis and inductive learning. It turns out that even a seemingly intangible phenomenon like musical expression can be transformed into something that can be studied formally, and that the computer can indeed discover some fundamental (and sometimes surprising) principles underlying the art of music performance. S. Lange, K. Satoh, and C.H. Smith (Eds.): DS 2002, LNCS 2534, pp. 13–32, 2002. c Springer-Verlag Berlin Heidelberg 2002
14
G. Widmer
The purpose of this paper is to lay out the general research questions that guide the project, to summarize the most important results achieved so far (with an emphasis on the most recent and, in fact, still very speculative work), and to identify challenging open research questions for the field of knowledge discovery and discovery science. In the process, it will become clear that computer-based discovery in a ‘real-world’ domain like music is a complex, multi-stage process where each phase requires both intelligent data analysis techniques and creativity on the part of the researcher. The title of the paper refers to the late Vladimir Horowitz, one of the most famous pianists of the 20th century, who symbolizes like few others the fascination that great performers hold for the general audience. Formally explaining the secret behind the art and magic of such a great master would indeed be an extremely thrilling feat. Needless to say, that is highly unlikely to be possible, and we freely admit that we chose Horowitz’s name mainly for metaphoric purposes, to attract the reader’s attention. The ‘Horowitz Factor’ will not be revealed here (though we have recently also started to work with Horowitz data). Still, we do hope the following description of project and results will capture the reader’s imagination. The paper is organized follows: section 2 presents the object of our investigations, namely, expressive music performance. Section 3 then gives an overview of the main lines of investigation currently pursued. Two different kinds of fundamental research questions will be identified there, and the two subsequent sections present our current results along these two dimensions. Section 4 is kept rather short (because these results have been published before). The main part of the paper is section 5, which describes a line of research currently under investigation (studing aspects of individual artistic style). Section 6 completes this report by trying to derive some general lessons for discovery science from our work so far, and by identifying a set of challenging research opportunities for further work in this area.
2
Expressive Music Performance
Expressive music performance is the art of shaping a musical piece by continuously varying important parameters like tempo, dynamics, etc. Human musicians do not play a piece of music mechanically, with constant tempo or loudness, exactly as written in the printed music score. Rather, they speed up at some places, slow down at others, stress certain notes or passages by various means, and so on. The most important parameter dimensions available to a performer (a pianist, in particular) are tempo and continuous tempo changes, dynamics (loudness variations), and articulation (the way successive notes are connected). Most of this is not specified in the written score, but at the same time it is absolutely essential for the music to be effective and engaging. The expressive nuances added by an artist are what makes a piece of music come alive (and what makes some performers famous).
In Search of the Horowitz Factor
15
Expressive variation is more than just a ‘distortion’ of the original (notated) piece of music. In fact, the opposite is the case: the notated music score is but a small part of the actual music. Not every intended nuance can be captured in a limited formalism such as common music notation, and the composers were and are well aware of this. The performing artist is an indispensable part of the system, and expressive music performance plays a central role in our musical culture. That is what makes it a central object of study in the field of musicology. Our approach to studying this phenomenon is data-driven: we collect recordings of performances of pieces by skilled musicians,1 measure aspects of expressive variation (e.g., the detailed tempo and loudness changes applied by the musicians), and search for patterns in these tempo, dynamics, and articulation data. The goal is interpretable models that characterize and ‘explain’ consistent regularities and patterns (if such should indeed exist). As we will see, that requires methods and algorithms from machine learning, data mining, pattern recognition, but also novel methods of intelligent music processing. Our research is meant to complement recent work (and the set of methods used) in contemporary musicology, which has largely been hypothesis-driven (e.g., [8,9,26,27,28,37]), although some researchers have also taken real data as the starting point of their investigations (e.g., [20,22,23,24]). In any case, our investigations are the most data-intensive empirical studies ever performed in the area of musical performance research.
3
Project Overview
Our starting point for the following presentation are two generic types of research questions regarding expressive music performance. First, are there general, fundamental principles of music performance that can be discovered and characterized? Are there general (possibly unconscious and definitely unwritten) rules that all or most performers adhere to? In other words, to what extent can a performer’s expressive actions be predicted? And second, is it possible to formally characterize and quantify aspects of individual artistic style? Can we desribe formally what makes the special art of a Vladimir Horowitz, for instance? The first set of questions thus relates to similarities or commonalities between different performances and different performers, while the second focuses on the differences. The following project presentation is structured according to these two types of questions. Section 4 focuses on the commonalities and briefly recapitulates some of our recent work on learning general performance rules from data. The major part of this report is section 5, which describes currently ongoing (and very preliminary) work on the discovery of stylistic characteristics of great artists. Both of these lines of research are complex enterprises and comprise a number of important steps — from the acquisition and measuring of pertinent data to computer-based discovery proper. The presentation in chapters 4 and 5 will be 1
At the moment, we restrict ourselves to classical tonal music, and to the piano.
16
G. Widmer
Starting Point: The Mystery of Expressive Music Performance
2 types of questions: commonalities / fundamental principles? -> Section 4 data collection / data extraction from MIDI (4.1) discovery of noteĆlevel rules (4.2)
data collection / data extraction from audio (5.1)
learning of multiĆlevel strategies (4.3)
integrated multiĆlevel model
differences / individual artistic style? -> Section 5
visualization of individual cases - The Worm! (5.2) transformation (segmentation & clustering) (5.3)
visualization of global characteristics (5.3)
string analysis: characteristic patterns (5.4)
The `Horowitz Factor'???
Fig. 1. Structure of project work presented in this paper (with pointers to relevant sections).
structured according to these main steps (see Figure 1). This presentation should give the reader an impression of the complexity of such a discovery process, and the efforts involved in each of the steps of the process.
4
Studying Commonalities: Searching for Fundamental Principles of Music Performance
The question we turn to first is the search for commonalities between different performances and performers: are there consistent patterns that occur in many performances and point to fundamental underlying principles? We are looking for general rules of music performance, and the methods used will come from the area of inductive machine learning. This section is kept rather short and only points to the most important results, because most of this work has already been published elsewhere [32,34, 35,36].
In Search of the Horowitz Factor 1.5 1.4 1.3 1.2 1.1 1 0.9 0.8 0.7 0.6
17
Dynamics Pianist 1 Dynamics Pianist 2 Dynamics Pianist 3
Fig. 2. Dynamics curves (relating to melody notes) of performances of the same piece (Fr´ed´eric Chopin, Etude op.10 no.3, E major) by three different Viennese pianists (computed from recordings on a B¨ osendorfer 290SE computer-monitored grand piano).
4.1
Data Acquisition: Measuring Expressivity in Performances
The first problem is data acquisition. What is required are precise measurements of the tempo, timing, dynamics, and articulation in a performance of a piece by a musician. In principle, we need to measure exactly when and how long and how loud each individual note was played, and how this deviated from the nominal values ‘prescribed’ in the written music score. Extracting this information with high precision from sound recordings is not possible, for basic signal processing reasons. Instead, our main source of information are special pianos that precisely record each action by a performer. In particular, the B¨ osendorfer 290SE is a full concert grand piano with a special mechanism that measures every key and pedal movement with high precision, and stores this information in a format similar to MIDI. (The piano also features a mechanical reproduction facility that can re-produce a recorded performance with very high accuracy.) From these measurements, and from a comparison of these with the notes as specified in the written score, every expressive nuance applied by a pianist can be computed. These nuances can be represented as expression curves. For instance, Figure 2 shows dynamics curves — the dynamics patterns produced by three different pianists in performing the same piece. More precisely, each point represents the relative loudness with which a particular melody note was played (relative to an averaged ‘standard’ loudness); a purely mechanical, unexpressive rendition of the piece would correspond to a perfectly flat horizontal line at y = 1.0. Variations in tempo and articulation can be represented in an analogous way. Figure 2 exhibits some clear common patterns and tendencies in the three performances. Despite individual differences between the recordings, there seem to be common strategies or ‘rules’ that are followed by the pianists, consciously or unconsciously. Obviously, there is hope for automated discovery algorithms to find some general principles.
18
G. Widmer
4.2
Induction of Note-Level Performance Rules
And some such general principles have indeed been discovered, with the help of a new inductive rule learning algorithm named PLCG [34]. When trained on a large set of example performances (13 complete piano sonatas by W.A.Mozart, played on the B¨ osendorfer 290SE by the Viennese concert pianist Roland Batik — that is some four hours of music and more than 106.000 played notes), PLCG discovered a small set of 17 quite simple classification rules that predict a surprisingly large number of the note-level choices of the pianist.2 The rules have been published in the musicological literature [35] and have created quite a lot of interest in the musicology world. The surprising aspect is the high number of note-level actions that can be predicted by very few (and mostly very simple) rules. For instance, four rules were discovered that together correctly predict almost 23% of all the situations where the pianist lengthened a note relative to how it was notated (which corresponds to a local slowing down of the tempo).3 To give the reader an impression of the simplicity and generality of the discovered rules, here is an extreme example: RULE TL2: abstract duration context = equal-longer & metr strength ≤ 1 ⇒ lengthen “Given two notes of equal duration followed by a longer note, lengthen the note (i.e., play it more slowly) that precedes the final, longer one, if this note is in a metrically weak position (‘metrical strength’ ≤ 1).” This is an extremely simple principle that turns out to be surprisingly general and precise: rule TL2 correctly predicts 1,894 cases of local note lengthening, which is 14.12% of all the instances of significant lengthening observed in the training data. The number of incorrect predictions is 588 (2.86% of all the counterexamples), which gives a precision (percentage of correct predictions) of .763. It is remarkable that one simple principle like this is sufficient to predict such a large proportion of observed note lengthenings in a complex corpus such as Mozart sonatas. This is a truly novel discovery; none of the existing theories of expressive performance were aware of this simple pattern. 2
3
By ‘note-level’ rules we mean rules that predict how a pianist is going to play a particular note in a piece — slower or faster than notated, louder or softer than its predecessor, staccato or legato. This should be contrasted with higher-level expressive strategies like the shaping of an entire musical phrase (e.g., with a gradual slowing down towards the end) — that will be addressed in section 4.3. It should be clear that a coverage of close to 100% is totally impossible, not only because expressive music performance is not a perfectly deterministic, predictable phenomenon, but also because the level of individual notes is clearly insufficient as a basis for a complete model of performance; musicians think not (only) in terms of single notes, but also in terms of higher-level musical units such as motifs and phrases — see section 4.3.
In Search of the Horowitz Factor
19
Moreover, experiments revealed that most of these rules are highly general and robust: they carry over to other performers and even music of different styles with virtually no loss of coverage and precision. In fact, when the rules were tested on performances of quite different music (Chopin), they exhibited significantly higher coverage and prediction accuracy than on the original (Mozart) data they had been learned from. What the machine has discovered here really seem to be fundamental performance principles. A detailed discussion of the rules as well as a quantitative evaluation of their coverage and precision can be found in [35]; the learning algorithm PLCG is described and analyzed in [34]. 4.3
Multi-level Learning of Performance Strategies
As already mentioned, not all of a performer’s decisions regarding tempo or dynamics can be predicted on a local, note-to-note basis. Musicians understand the music in terms of a multitude of more abstract patterns and structures (e.g., motifs, groups, phrases), and they use tempo and dynamics to ‘shape’ these structures, e.g., by applying a gradual crescendo (growing louder) or decrescendo (growing softer) to entire passages. In fact, music performance is a multi-level phenomenon, with musical structures and performance patterns at various levels embedded in each other. Accordingly, the set of note-level performance rules described above is currently being augmented with a multi-level learning strategy where the computer learns to predict elementary tempo and dynamics ‘shapes’ (like a gradual crescendo-decrescendo) at different levels of the hierarchical musical phrase structure, and combines these predictions with local timing and dynamics modifications predicted by learned note-level models. Preliminary experiments, again with performances of Mozart sonatas, yielded very promising results [36]. Just to give an idea, Figure 3 shows the predictions of the integrated learning algorithm on a test piece after learning from other Mozart sonatas. As can be seen in the lower part of the figure, the system manages to predict not only local patterns, but also higher-level trends (e.g., gradual increases of overall loudness) quite well. Sound examples will be made available at our project home page. (http://www.oefai.at/music). The ultimate goal of this work is an integrated model of expressive performance that combines note-level rules with structure-level strategies. Our current work along these lines focuses on new learning algorithms for discovering and predicting patterns that are hierarchically related to each other.
5
Studying Differences: Trying to Characterize Individual Artistic Style
The second set of questions guiding our research concerns the differences between individual artists. Can one characterize formally what is special about the style of a particular pianist? Contrary to the research on common principles described
20
G. Widmer
rel. dynamics
1.5
1
0.5 25
30
35 40 abs. score position
45
50
25
30
35 40 abs. score position
45
50
rel. dynamics
1.5
1
0.5
Fig. 3. Learner’s predictions for the dynamics curve of Mozart Sonata K.280, 3rd movement, mm. 25–50. Top: dynamics shapes predicted for phrases at four levels; bottom: composite predicted dynamics curve resulting from phrase-level shapes and note-level predictions (grey) vs. pianist’s actual dynamics (black). Line segments at the bottom of each plot indicate hierarchical phrase structure.
in section 4 above, where we used mainly performances by local (though highly skilled) pianists, here we are explicitly interested in studying famous artists. Can we find the ‘Horowitz Factor’ ? This may be the more intriguing question to the general audience, because it involves famous artists. However, the reader must be warned that this is a very difficult question. The following is work in progress, the current results are highly uncertain and incomplete, and the examples given below should be taken as indications of the kinds of things that we hope to discover, rather than truly significant discovery results. 5.1
Data Acquisition: Measuring Expressivity in Audio Recordings
The first major difficulty is data aquisition. With famous pianists, the only source of data are audio recordings, i.e., records and music CDs (we cannot very well invited them all to Vienna to perform on the B¨ osendorfer 290SE piano). Unfortunately, it is impossible, with current signal processing methods, to extract
In Search of the Horowitz Factor
21
precise performance information (start and end times, loudness, etc.) about each individual note directly from audio data. Thus, it will not be possible to perform studies at the same level of detail as those based on MIDI data. In particular, we cannot study how individual notes are played. What is currently possible is to extract tempo and dynamics at the level of the beat.4 That is, we extract those time points from the audio recordings that correspond to beat locations. From the (varying) time intervals between these points, the beat-level tempo and its changes can be computed. Beat-level dynamics is also computed from the audio signal as the overall loudness/amplitude of the signal at the beat times. The hard problem here is automatically detecting and tracking the beat in audio recordings. Indeed, this is an open research problem that forced us to develop a novel beat tracking algorithm [4]. Beat tracking, in a sense, is what human listeners do when they listen to a piece and tap their foot in time with the music. As with many other perception and cognition tasks, what seems easy and natural for a human turns out to be extremely difficult for a machine. The main problems to be solved are (a) detecting the onset times of musical events (notes, chords, etc.) in the audio signal, (b) deciding which of these events carry the beat (that includes determining the basic tempo, i.e., the basic rate at which beats are expected to occur), and (c) tracking the beat through tempo changes. The latter part is extremely difficult in classical music, where the performer may change the tempo drastically — a slowing down by 50% within one second is nothing unusual. It is difficult for a machine to decide whether an extreme change in inter-beat intervals is due to the performer’s expressive timing, or whether it indicates that the algorithm’s beat hypothesis was wrong. Experimental evaluations showed that our beat tracking algorithm is probably among the best currently available [6]. In systematic experiments with 13 Mozart sonatas, the algorithm achieved a correct tracking rate of over 90%. However, for our investigations we need a tracking accuracy of 100%, so we opted for a semi-automatic, interactive procedure. The beat tracking algorithm was integrated into an interactive computer program5 that takes a piece of music (a sound file), tries to track the beat, displays its beat hypotheses visually on the screen, allows the user to listen to (selected parts of) the tracked piece and modify the beat hypothesis by adding, deleting, or moving beat indicators, and then attempts to re-track the piece based on the updated information. This is still a very laborious process, but it is much more efficient than ‘manual’ beat tracking. 4
5
The beat is an abstract concept related to the metrical structure of the music; it corresponds to a kind of quasi-regular pulse that is perceived as such and that structures the music. Essentially, the beat is the time points where listeners would tap their foot along with the music. Tempo, then, is the rate or frequency of the beat and is usually specified in terms of beats per minute. The program has been made publicly available and can be downloaded from http://www.oefai.at/˜simon.
22
G. Widmer
After a recording has been processed in this way, tempo and dynamics at the beat level can be easily computed. That is the input data to the next processing step. 5.2
Data Visualization: The Performance Worm
To facilitate visual analysis and an intuitive musical interpretation of the tempo and dynamics measurements, a new representation and, based on that, a visualization system were developed, using a recent idea and method by the German musicologist J¨org Langner [13]. The two basic ideas are to integrate tempo and dynamics into a single representation, and to compute smoothed trajectories from the beat-level measurements. Given a series of measurements points (tempo or loudness values) over time, the series is smoothed using overlapping Gaussian windows. The motivation for this choice (which, among other things, has to do with issues of musical and visual perception) and the precise method are described in [14]. The smoothed sequences of tempo and dynamics values are then combined into a joint sequence of pairs of coordinates, which represent the development of tempo and dynamics over time as a trajectory in a two-dimensional space, where tempo and dynamics define the axes, and the time dimension is implicit in the trajecory. The method has been implemented in a interactive visualization system that we call the Performance Worm [7]. The Worm can play a given
Fig. 4. Snapshot of the Performance Worm at work: First four bars of Daniel Barenboim’s performance of Mozart’s F major sonata K.332, 2nd movement. Horizontal axis: tempo in beats per minute (bpm); vertical axis: loudness in sone (a psycho-acoustic measure [38]). Movement to the upper right indicates a speeding up (accelerando) and loudness increase (crescendo) etc. etc. The darkest points represents the current instant, while instants further in the past appear fainter.
In Search of the Horowitz Factor
23
35
30
25
20
15
10
5
0 10
15
20
25
30
35
40
Fig. 5. A complete worm: smoothed tempo-loudness trajectory representing a performance of Mozart’s F major sonata K.332, 2nd movement, by Mitsuko Uchida. Horizontal axis: tempo in beats per minute (bpm); vertical axis: loudness in sone [38].
recording and, in parallel, show the movement of the trajectory in an animated display. Figure 4 shows an example of the Worm at work. This is a very intuitive representation that facilitates a direct appreciation and understanding of the performance strategies applied by different pianists. The subsequent analysis steps will be based on this representation. We are currently investing large efforts into measuring tempo and dynamics in recordings of different pieces by different famous pianists, using the interactive beat tracking system mentioned above. One such data collection that will be referred to in the following consists of five complete piano sonatas by W.A.Mozart (54 separate sections), as played by the five pianists Daniel Barenboim, Glenn Gould, Maria Jo˜ ao Pires, Andr´ as Schiff, and Mitsuko Uchida. Beat-tracking these 54 × 5 recordings took roughly two person months. Figure 5 shows a complete tempo-loudness trajectory representing a performance of one movement of a piano sonata by Mitsuko Uchida. 5.3
Transforming the Problem: Segmentation, Clustering, Cluster Visualization
Instead of analyzing the raw tempo-loudness trajectories directly, we chose to pursue an alternative route, namely, to transform the data representation and thus the entire discovery problem into a form that is accessible to common inductive learning and data mining algorithms. To that end, the performance trajectories are segmented into short segments of fixed length (e.g., 2 beats), these segments are optionally subjected to various normalization operations (e.g., mean and/or variance normalization to abstract away from absolute tempo and loudness and/or absolute pattern size, respectively). The resulting segments are
24
G. Widmer
Fig. 6. A ‘Mozart performance alphabet’ (cluster prototypes) computed by segmentation and clustering from performances of Mozart piano sonatas by five famous pianists (Daniel Barenboim, Glenn Gould, Maria Jo˜ ao Pires, Andr´ as Schiff, Mitsuko Uchida). To indicate directionality, dots mark the end points of segments.
then grouped into classes of similar patterns via clustering. For each of the resulting clusters, a prototype (centroid) is computed. These prototypes represent a set of typical elementary tempo-loudness patterns that can be used to approximately reconstruct a ‘full’ trajectory (i.e., a complete performance). In that sense, they can be seen as a simple alphabet of performance (restricted to tempo and dynamics). Figure 6 shows a set of prototypical patterns computed from the above-mentioned set of Mozart sonata performances. The number of alphabets one could compute is potentially infinite, due to a multitude of existing clustering algorithms with their parameters (in particular, most clustering algorithms require an a priori specification of the number of desired clusters) and possible normalization operations. In order to reduce the number of degrees of freedom (and arbitrary decisions to be made), extensive research is currently under way on non-parametric methods for determining an optimal number of clusters. Particular candidates are information-theoretic approaches based on the notion of minimum encoding length [19,15]. The particular clustering shown in Figure 6 was generated by a SelfOrganizing Map (SOM) algorithm [12]. A SOM produces a geometric layout of the clusters on a two-dimensional grid or map, attempting to place similar clusters close to each other. That property facilitates a simple, intuitive visualization method. The basic idea, named Smoothed Data Histograms (SDH), is to visualize the cluster distribution in a given data set by estimating the probability density of the high-dimensional data on the map (see [21] for details). Figure 7 shows how this can be used to visualize the frequencies with which certain pi-
In Search of the Horowitz Factor
25
Fig. 7. Visualization, based on ‘smoothed data histograms’, of the distribution of stylistic patterns in Mozart performances by four pianists (see Figure 6 for the corresponding cluster map). Bright areas indicate high density.
anists use elementary expressive patterns (trajectory segments) from the various clusters. The differences are striking: the most common stylistic elements of, say, a Daniel Barenboim seem to be quite different from those of Andr´ as Schiff, for instance. 5.4
Structure Discovery in Musical Strings
The SDH cluster visualization method gives some insight into very global aspects of performance style; it does show that there are indeed significant differences between the pianists. But we want to get more detailed insight into characteristic patterns and performance strategies. To that end, another (trivial) transformation is applied to the data. We can take the notion of an alphabet literally and associate each prototypical elementary tempo-dynamics shape (i.e., each cluster prototype) with a letter. A full performance — a complete trajectory in tempo-dynamics space — can be approximated by a sequence of elementary prototypes, i.e., a sequence of letters, i.e., a string. Figure 8 shows a part of a performance of a Mozart sonata movement, coded in terms of such an alphabet. This final transformation step, trivial as it may be, makes it evident that our original musical problem has now been transferred into a quite different world: the world of string analysis. The fields of pattern recognition, machine learning, data mining, etc. have developed a rich set of methods that can find structure in strings and that could now profitably be applied to our musical data.6 6
However, it is also clear that through this long sequence of transformation steps — smoothing, segmentation, normalization, replacing individual elementary patterns by a prototype — a lot of information has been lost. It is not clear at this point whether this reduced data representation still permits truly significant discoveries.
26
G. Widmer PSWPKECSNRVTORRWTTOICXPMATPGFRQIFBDHDTPQIEEFECDLGCSQIEETPOVTHGA RQMEECDSNQQRTPSMFATLHHDXTOIFEATTPLGARWTPPLHRNMFAWHGARQMCDHFAQHH DNIERTPKFRTPPPKFRWPRWPLRQVTOECSQDJRVTTPQICURWDSKFERQICDHDTTPQEE CGFEWPIEXPMFEEXTTTHJARQARCDSNQEUVTPNIEXPHHDDSRQIFECXTPOMAVTHDTP HDSPQFAVHDURWHJEEFBHGFEFARVPLARWTPNIEVTPPTPHRMEARIEXPPTTOQICDIB TLJBJRRRWSRVTTPLTPNIEETPPXOIFEEECDORTTOQMBURWHFBDTPKFECHJFRTKUT PMFETORRLXPNQCTPPXSIEBDHHDNRQICDSQEDTTOMCDJQIEECSOTLHDNMARVPPTN QUVMIBDTPNQWTTLURQIEFEASRTPLJFEEUQBRVTORTOIEAMEDTPPHDNQIDGEHJRR VTLGARQCHGAVPNRVHDURWTPQIEUVLGFECDDPNQIEEXHHDJFFEURTTLGFEEANQBS .............................
Fig. 8. Beginning of a performance by Daniel Barenboim (W.A.Mozart, piano sonata K.279 in C major) coded in terms of a simple 24 letter ‘performance alphabet’ derived from a clustering of performance trajectory segments.
There is a multitude of questions one might want to ask of these musical strings. For instance, one might search for local patterns (substrings) that are characteristic of a particular pianist (that will be demonstrated below). One might search for general, frequently occurring substrings (i.e., more extended performance patterns) that are typical components of performances (‘stylistic clich´es’, so to speak). Using such frequent patterns as building blocks, one might try to use machine learning algorithms to induce (partial) grammars of musical performance style, both deterministic (e.g., [18]) or probabilistic (e.g,. [2]). A broad range of methods is available. Whether their application produces results that make musical sense remains to be investigated. A First Step: Discovering Discriminative Frequent Substrings. As a first simple example, let us consider the following question: are there substrings in these musical performance strings that exhibit a certain minimum frequency and that are characteristic of a particular pianist? In data mining terms, these could be called discriminative frequent sequences. Assume we have p pianists. If we represent each pianist by one string built by concatenating all the pianist’s performances, the problem can be stated as follows: let S = {S1 , ..., Sp } be a set of strings (each representing a pianist), and let occ(Xi , Sj ) denote the set of occurrences ofsome character sequence Xi in p a string Sj ; let nij = |occ(Xi , Sj )| and Ni = j=1 nij (i.e., the total number of occurrences of Xi over all strings Sj ∈ S). Given a pre-defined minimum frequency threshold Θ, the goal then is to find a set of character sequences X = {Xi : ∃j [ nij ≥ Θ ∧ nik = 0 ∀k = j ]},
(1)
that is, sequences with a certain minimum frequency that occur only in one string (i.e., in performances by one pianist). In reality, sequences that perfectly single In any case, whatever kinds of patterns may be found in this representation will have to be tested for musical significance in the original data.
In Search of the Horowitz Factor
27
out one pianist from the others will be highly unlikely, so instead of requiring uniqueness of a pattern to a particular pianist, we will be searching for patterns that exhibit a certain level of disciminatory power; let us call these approximately discriminative frequent sequences. As a measure of discriminatory power of a sequence we will use an entropy-like measure (see below). Approximately discriminative frequent sequences can be found by a simple two-stage algorithm. In the first step, all (maximal) sequences that are frequent overall (i.e., all Xi with Ni > Θ) are found by a straightforward version of the exhaustive level-wise search that is used in data mining for the discovery of association rules [1] and frequent episodes [16]. In the second step, the frequent sequences are sorted in ascending order according to the ‘entropy’ of their distribution over the pianists, as defined in Equation 2:
E(Xi ) =
p j=1
−
nij nij · log2 , Ni Ni
(2)
and then we simply select the first k (the ones with the lowest entropy) from the sorted list as the most discriminative ones. These will tend to be patterns that occur frequently in one or few pianists, and rarely in the others. As we currently have to inspect (and listen to) all these patterns in order to find out which ones are musically interesting (we have no operational mathematical definition of musical ‘interestingness’), this crude method of ‘look at the k most discriminative ones’ is good enough for our current purposes. Also, first searching for all frequent sequences and then selecting the ones with highest discriminative power may seem an inefficient method, but is sufficient for our current situation with strings of only some tens of thousands of characters. It would be straightforward to devise a more efficient algorithm that uses a discriminativity (e.g., entropy) threshold for pruning the search for frequent sequences. In very preliminary experiments with our corpus of Mozart performances (5 sonatas, 54 sections, 5 pianists), coded in a variety of different ‘performance alphabets’, a number of sequences were discovered that appear to be discriminative (at least in our limited corpus of data) and also look like they might be musically interesting. For example, in the alphabet used in the encoding of Figure 8, the sequence FAVT came up as a typical Barenboim pattern, with 7 occurrences in Barenboim’s Mozart performances, 2 in Pires, 1 in Uchida, and 0 in Schiff and Gould. To find out whether such a sequence codes any musically interesting or interpretable behaviour, we can go back to the original data (the tempo-loudness trajectories) and identify the segments of the trajectories coded by the various occurrences of the sequence. As Figure 9 (left part) shows, what is coded by the letter sequence FAVT in Daniel Barenboim’s performances of Mozart is an increase in loudness (a crescendo), followed by a slight tempo increase (accelerando), followed by a decrease in loudness (decrescendo) with more or less constant tempo. That is indeed a rather unusual pattern. In our experience so far, it is quite rare to see a pianist speed up during a loudness maximum. Much
28
G. Widmer FAVT barenboim t2_norm_mean_var (7)
SC uchida t4_norm_mean (8) 40
26
24
35
22
Loudness [sone]
Loudness [sone]
30 20
18
25
20 16
15
14
12 50
100
150
200 Tempo [bpm]
250
300
350
10 70
80
90
100
110 Tempo [bpm]
120
130
140
150
Fig. 9. Two sets of (instantiations of) performance patterns: FAVT sequence typical of Daniel Barenboim (left) and SC pattern (in a different alphabet) from Mitsuko Uchida (right). To indicate directionality, a dot marks the end point of a segment.
more common in such situations are slowings down (ritardandi), which gives a characteristic counterclockwise movement of the Worm (as, e.g., in the right half of Figure 9, which shows instantiations of a pattern that seems characteristic of the style of Mitsuko Uchida (8 occurrences vs. 0 in all the other pianists)). But before being carried away by musical interpretations and hypotheses, we must remind ourselves once more that the current data situation is too limited to draw serious conclusions, and determining the musical significance of such patterns will be a complex problem. The absolute numbers (7 or 8 occurrences of a supposedly typical pattern in a pianist) are too small to support claims regarding statistical significance. Also, we cannot say with certainty that similar patterns do not occur in the performances by the other pianists just because they do not show up as substrings there — they might be coded by a slightly different character sequence! And finally, many alternative performance alphabets could be computed; we currently have no objective criteria for determining the optimal one in any sense. So the reader is cautioned not to take any of the patterns shown here too literally. They are only indicative of the kinds of things we hope to discover with our methods in the (near) future. Whether these findings will indeed be musically relevant can only be hoped for at the moment.
6
Conclusions
Expressive music performance is a complex phenomenon, and what has been achieved and discovered so far are only tiny parts of a big mosaic. Still, we do feel that the project has already produced a number of results that are interesting and justify this computer-based discovery approach. Regarding the induction of note-level rules (section 4.2), it seems safe to say that this is the first time a machine has made significant discoveries in music. Some of the rules were new to musicologists and complement other rule-based
In Search of the Horowitz Factor
29
musicological models of performance [8]. Extending the rule set with multi-level learning of more abstract patterns, as indicated in section 4.3, will lead to an operational model that can actually produce musically sensible interpretations (whether that has any practical applications is currently open). And our preliminary work on performance visualization and pattern extraction (sections 5.2, 5.3, and 5.4) does seem to indicate that it will also be possible to get completely new kinds of insight into the style of great artists. Along the way, a number of novel methods of potentially general benefit were developed: the beat tracking algorithm and the interactive tempo tracking system [4], the Performance Worm [7] (with possible applications in music education and analysis), and the PLCG rule learning algorithm [34]. Also, we have compiled what is most likely the largest database of high-quality performance measurements, both in the form of MIDI measurements and beat-tracked sound recordings. Such data collections will be an essential asset in future research. However, the list of limitations and open problems is much longer, and it seems to keep growing with every step forward we take. One limiting factor is the measuring problem. Currently, it is only possible to extract rather crude and global information from audio recordings; we cannot get at details like timing, dynamics, and articulation of individual voices or individual notes. This is a challenge for signal analysis and music processing, and we are indeed working on improved intelligent expression extraction methods. On the data anlysis and discovery side, the preliminary pattern discovery work described in section 5.4 above has only looked at the strings (i.e., the performance trajectories) themselves, without regard to the underlying music. Obviously, the next step must be to search for systematic relations between patterns in the performance trajectories, and the content and structure of the music that is being played. Also, it is clear that most of the secrets of the great masters’ style — if they are at all describable — will not be hidden in the kinds of local patterns that were just described. Important aspects of structural music interpretation and high-level performance strategies will only become visible at higher abstraction levels. These questions offer to be a rich source of challenges for sequence analysis and pattern discovery. Things we are currently thinking about include generalised notions of frequent sequences (e.g., based on a notion of graded letter similarity), methods for discovering frequent subpatterns directly from the numeric time series, unsupervised learning of useful abstract building blocks from sequences, algorithms for inducing partial grammars (which may be allowed to account for only parts of a string), methods for learning the connection between grammar rules and properties of the underlying music, and methods for decomposing complex trajectories into more elementary patterns at different structural levels, which may have different kinds of ‘explanations’ and thus require different models. We should also look into work on related phenomena in language, specifically, prosody, which might offer useful methods for our musical problem.
30
G. Widmer
In view of all this, this project will most likely never be finished. But much of the beauty of research is in the process, not in the final results, and we do hope that our sponsors share this view and will keep supporting what we believe is an exciting research adventure. Acknowledgements. The project is made possible by a very generous START Research Prize by the Austrian Federal Government, administered by the Austrian Fonds zur F¨ orderung der Wissenschaftlichen Forschung (FWF) (project no. Y99-INF). Additional support for our research on AI, machine learning, scientific discovery, and music is provided by the European project HPRN-CT2000-00115 (MOSART) and the EU COST Action 282 (Knowledge Exploration in Science and Technology). The Austrian Research Institute for Artificial Intelligence acknowledges basic financial support by the Austrian Federal Ministry for Education, Science, and Culture. Thanks to Johannes F¨ urnkranz for his implementation of the level-wise search algorithm. I am indebted to my project team Simon Dixon, Werner Goebl, Elias Pampalk, and Asmir Tobudic for all their great work.
References 1. Agrawal, R., and Srikant, R. (1994). Fast Algorithms for Mining Association Rules. In Proceedings of the 20th International Conference on Very Large Data Bases (VLDB), Santiago, Chile. 2. Chen, S.F. (1995). Bayesian Grammar Induction for Language Modeling. In Proceedings of the 33rd Annual Meeting of the Association for Computational Linguistics, pp.228–235. 3. Dietterich, T. G. (2000). Ensemble Methods in Machine Learning. In J. Kittler and F. Roli (Ed.), First International Workshop on Multiple Classifier Systems. New York: Springer Verlag. 4. Dixon, S. (2001). Automatic Extraction of Tempo and Beat from Expressive Performances. Journal of New Music Research 30(1), 39–58. 5. Dixon, S. and Cambouropoulos, E. (2000). Beat Tracking with Musical Knowledge. In Proceedings of the 14th European Conference on Artificial Intelligence (ECAI2000), IOS Press, Amsterdam. 6. Dixon, S. (2001). An Empirical Comparison of Tempo Trackers. In Proceedings of the VIII Brazilian Symposium on Computer Music (SBCM’01), Fortaleza, Brazil. 7. Dixon, S., Goebl, W., and Widmer, G. (2002). The Performance Worm: Real Time Visualisation of Expression Based on Langner’s Tempo-Loudness Animation. In Proceedings of the International Computer Music Conference (ICMC’2002), G¨ oteborg, Sweden. 8. Friberg, A. (1995). A Quantitative Rule System for Musical Performance. Doctoral Dissertation, Royal Institute of Technology (KTH), Stockholm, Sweden. 9. Gabrielsson, A. (1999). The Performance of Music. In D. Deutsch (Ed.), The Psychology of Music (2nd ed., pp. 501-602). San Diego: Academic Press. 10. Hunter, L. (ed.) (1993). Artificial Intelligence and Molecular Biology. Menlo Park, CA: AAAI Press.
In Search of the Horowitz Factor
31
11. King, R.D., Muggleton, S., Lewis, R.A., and Sternberg, M.J.E. (1992). Drug Design by Machine Learning: The Use of Inductive Logic Programming to Model the Structure-activity Relationship of Trimethoprim Analogues Binding to Dihydrofolate Reductase. In Proceedings of the National Academy of Sciences, Vol. 89, pp.11322–11326. 12. Kohonen, T. (2001). Self-Organizing Maps, 3rd edition. Berlin: Springer Verlag. 13. Langner, J. and Goebl, W. (2002). Representing Expressive Performance in TempoLoudness Space. In Proceedings of the ESCOM Conference on Musical Creativity, Li`ege, Belgium . 14. Langner, J., and Goebl, W. (2002). Visualizing Expressive Performance in TempoLoudness Space. Submitted to Computer Music Journal. Preliminary version available on request from the author. 15. Ludl, M. and Widmer, G. (2002). Towards a Simple Clustering Criterion Based on Minimum Length Encoding. In Proceedings of the 13th European Conference on Machine Learning (ECML’02), Helsinki. Berlin: Springer Verlag. 16. Mannila, H., Toivonen, H., and Verkamo, I. (1997). Discovery of Frequent Episodes in Event Sequences. Data Mining and Knowledge Discovery, 1(3), 259–289. 17. Muggleton, S., King, R.D., and Sternberg, M.J.E. (1992). Protein Secondary Structure Prediction Using Logic-based Machine Learning. Protein Engineering 5(7), 647–657. 18. Nevill-Manning, C.G. and Witten, I.H. (1997). Identifying Hierarchical Structure in Sequences: A linear-time algorithm. Journal of Artificial Intelligence Research 7, 67-82. 19. Oliver, J., Baxter, R., and Wallace, C. (1996). Unsupervised Learning Using MML. In Proceedings of the 13th International Conference on Machine Learning (ICML’96). San Francisco, CA: Morgan Kaufmann. 20. Palmer, C. (1988). Timing in Skilled Piano Performance. Ph.D. Dissertation, Cornell University. 21. Pampalk, E., Rauber, A., and Merkl, D. (2002). Using Smoothed Data Histograms for Cluster Visualization in Self-Organizing Maps. In Proceedings of the International Conference on Artificial Neural Networks (ICANN’2002), Madrid. 22. Repp, B. (1992). Diversity and Commonality in Music Performance: An Analysis of Timing Microstructure in Schumann’s ‘Tr¨ aumerei’. Journal of the Acoustical Society of America 92(5), 2546–2568. 23. Repp, B. H. (1998). A Microcosm of Musical Expression: I. Quantitative Analysis of Pianists’ Timing in the Initial Measures of Chopin’s Etude in E major. Journal of the Acoustical Society of America 104, 1085–1100. 24. Repp, B. H. (1999). A Microcosm of Musical Expression: II. Quantitative Analysis of Pianists’ Dynamics in the Initial Measures of Chopin’s Etude in E major. Journal of the Acoustical Society of America 105, 1972–1988. 25. Shavlik, J.W., Towell, G., and Noordewier, M. (1992). Using Neural Networks to Refine Biological Knowledge. International Journal of Genome Research 1(1), 81–107. 26. Sundberg, J. (1993). How Can Music Be Expressive? Speech Communication 13, 239–253. 27. Todd, N. (1989). Towards a Cognitive Theory of Expression: The Performance and Perception of Rubato. Contemporary Music Review, vol. 4, pp. 405–416. 28. Todd, N. (1992). The Dynamics of Dynamics: A Model of Musical Expression. Journal of the Acoustical Society of America 91, pp.3540–3550. 29. Vald´es-P´erez, R.E. (1995). Machine Discovery in Chemistry: New Results. Artificial Intelligence 74(1), 191–201.
32
G. Widmer
30. Vald´es-P´erez, R.E. (1996). A New Theorem in Particle Physics Enabled by Machine Discovery. Artificial Intelligence 82(1-2), 331–339. 31. Vald´es-P´erez, R.E. (1999). Principles of Human-Computer Collaboration for Knowledge Discovery in Science. Artificial Intelligence 107(2), 335-346. 32. Widmer, G. (2001). Using AI and Machine Learning to Study Expressive Music Performance: Project Survey and First Report. AI Communications 14(3), 149– 162. 33. Widmer, G. (2001). The Musical Expression Project: A Challenge for Machine Learning and Knowledge Discovery. In Proceedings of the 12th European Conference on Machine Learning (ECML’01), Freiburg, Germany. Berlin: Springer Verlag. 34. Widmer, G. (2001). Discovering Strong Principles of Expressive Music Performance with the PLCG Rule Learning Strategy. In Proceedings of the 12th European Conference on Machine Learning (ECML’01), Freiburg, Germany. Berlin: Springer Verlag. 35. Widmer, G. (2002). Machine Discoveries: A Few Simple, Robust Local Expression Principles. Journal of New Music Research 31(1) (in press). 36. Widmer, G., and Tobudic, A. (2002). Playing Mozart by Analogy: Learning Multilevel Timing and Dynamics Strategies. Journal of New Music Research (to appear). 37. Windsor, L. and Clarke, E. (1997). Expressive Timing and Dynamics in Real and Artificial Musical Performances: Using an Algorithm as an Analytical Tool. Music Perception 15, 127–152. 38. Zwicker, E. and Fastl, H. (2001). Psychoacoustics. Facts and Models. Springer Series in Information Sciences, Vol.22. Berlin: Springer Verlag.
Learning Structure from Sequences, with Applications in a Digital Library Ian H. Witten Department of Computer Science, University of Waikato Hamilton, New Zealand [email protected]
Abstract. The services that digital libraries provide to users can be greatly enhanced by automatically gleaning certain kinds of information from the full text of the documents they contain. This paper reviews some recent work that applies novel techniques of machine learning (broadly interpreted) to extract information from plain text, and puts it in the context of digital library applications. We describe three areas: hierarchical phrase browsing, including efficient methods for inferring a phrase hierarchy from a large corpus of text; text mining using adaptive compression techniques, giving a new approach to generic entity extraction, word segmentation, and acronym extraction; and keyphrase extraction.
The full version of this paper is published in the Proceedings of the 13th International Conference on Algorithmic Learning Theory, Lecture Notes in Artificial Intelligence Vol. 2533
S. Lange, K. Satoh, and C.H. Smith (Eds.): DS 2002, LNCS 2534, p. 33, 2002. c Springer-Verlag Berlin Heidelberg 2002
Discovering Frequent Structured Patterns from String Databases: An Application to Biological Sequences Luigi Palopoli and Giorgio Terracina DIMET - Universit` a di Reggio Calabria Localit` a Feo di Vito 89100 Reggio Calabria, Italy {palopoli,terracina}@ing.unirc.it
Abstract. In the last years, the completion of the human genome sequencing showed up a wide range of new challenging issues involving raw data analysis. In particular, the discovery of information implicitly encoded in biological sequences is assuming a prominent role in identifying genetic diseases and in deciphering biological mechanisms. This information is usually represented by patterns frequently occurring in the sequences. Because of biological observations, a specific class of patterns is becoming particularly interesting: frequent structured patterns. In this respect, it is biologically meaningful to look at both “exact” and “approximate” repetitions of the patterns within the available sequences. This paper gives a contribution in this setting by providing some algorithms which allow to discover frequent structured patterns, either in “exact” or “approximate” form, present in a collection of input biological sequences.
1
Introduction
A large number of text databases have been recently produced, as the result of both technological improvement and the growth of the Internet allowing for an “almost everywhere” accessibility of information. Unfortunately, the availability of such large amounts of raw data does not produce “per se” a necessary enlargement of the available “information”. This because such information is often implicitly encoded in the data. But making this information explicit is a non trivial task and, usually, it cannot be done by hand. This implies the need to devise automatic methods by which meaningful information can be extracted from string databases. In the last years, a particular class of raw data stored in string databases is assuming a prominent role in discovery science: genomic data. Indeed, the completion of the human genome sequencing showed up a wide range of new challenging issues involving sequence analysis. Genome databases mainly consists of sets of strings representing DNA or protein sequences (biosequences); unfortunately, most of these strings still require to be “interpreted”.
Corresponding author
S. Lange, K. Satoh, and C.H. Smith (Eds.): DS 2002, LNCS 2534, pp. 34–46, 2002. c Springer-Verlag Berlin Heidelberg 2002
Discovering Frequent Structured Patterns from String Databases
35
In the context of sequence analysis, pattern discovery has recently assumed a fundamental role. It deals with automatic methods for pattern and motif discovery in biosequences. The PROSITE Data Base [3] is very well fit to illustrate the notion of motif and the “discovery process” involved. It is a Data Base of protein families, where each family is described by a regular expression (the motif or pattern) highlighting important and common domains within the sequences in the given protein family. Unfortunately, those motifs have been extracted semi-automatically through the analysis of multiple alignments: a procedure no longer adequate, given the amount of biological data being produced worldwide. In fact, automatic pattern discovery is now more crucial than ever. The task of pattern discovery must be clearly distingueshed from that of matching a given pattern in a Data Base. Indeed, in the latter case, we know what we are looking for, while in the former we do not know. A particular class of patterns seems to be highly promising in the analysis of biosequence functions. Indeed, it has been observed [9] that, in complex organisms, the relative positions of substrings recognized as motifs are often not random when they participate in biological processes. For instance, the most frequently observed prokaryotic promoter regions are in general composed by two parts positioned approximately 10 and 35 bases respectively upstream from the start of transcription. The biological reason for this particular positions is that the two parts are recognized by the same protein. Analogously, in eukaryotic transcription regulation, different portions of sequences suitably positioned w.r.t. each other, promote or inhibit the transcription in presence of certain proteins. As a further example satellites and tandem repeats are repetitive structures, important to identify genetic markers and for their association to some genetic diseases [4,15]; extraction of patterns common to a set of sequences indicate conservation, which in turn indicates that the (sometimes unknown) function being described by the pattern is important [3,16,17]; promoter sequences and regulatory regions have a specific “high level” structure and their identification is also essential [7,8,13,14]. It is clear that discovering frequent structured patterns is therefore biologically meaningful and particularly important for the recognition of biological functions. As a further complication, looking for exact repetitions of patterns may lead to biologically unsatisfactory results. Indeed similar biological subsequences may have the same functions in their sequences. It is therefore mandatory to look for structured patterns “representing” a sufficiently high number of strings in the database. This problem is a nontrivial one [6,14] and requires high computational resources. In this paper we propose an efficient solution to the problem of discovering structured motifs in biological databases by exploiting Subword Trees [1,2,5, 10,11] as support index structures. In particular, we face four problems: (i) discovering simple structured motifs composed by two highly conserved regions separated by constrained spacers (exact repetitions) (Section 3); (ii) extending the previous solution for extracting complex structured motifs composed by
36
L. Palopoli and G. Terracina
r > 2 regions (Section 4); (iii) deriving structured motifs with the same structure as in (i) but allowing up to e errors in string repetitions (Section 5) and, finally (iv) discovering more complex structured motifs allowing up to e errors in string repetitions (Section 6). Before all that, in the following section, we provide some preliminary definitions that will be used throughout the paper.
2
Statement of the Problem
In biological contexts, a motif is a highly conserved region of nucleotide sequences. In other words, given a collection SC of N strings and a string m, m is called a motif for SC if it is repeated in at least q of the N strings in SC. The number q is called quorum. If we look for exact repetitions of m in SC we say that m is an exact motif. However, in some situations, exact repetitions do not suffice to represent biologically significant pieces of information because similar substrings may play the same role in biological processes. Therefore, it is often necessary to verify if a string m “represents” a set of strings occurring in SC. In this paper we consider that a string m represents a string m with an error e if the Hamming distance between m and m (denoted h(m, m )) is no larger than e1 ; m is said to be an e-occurrence of m. In this setting, a string m is called an e-motif if it has an e-occurrence in at least q of the N strings in SC. As pointed out in the introduction, it is biologically meaningful to search for frequent patterns structured as two or more parts separated by constrained spacers either in their exact form or allowing for errors in the match between the pattern and the strings. Before giving a formal definition of the problems we are going to analyze, it is necessary to define the notation exploited in the paper. Given an alphabet Σ we will denote single symbols in Σ with uppercase letters, whereas lowercase letters identify strings on Σ. The notation m = uv indicates that the string m is obtained by concatenating the (sub)strings u and v. The notation s[k] is used to indicate that the string s has length k. The special symbol X, not present in Σ, is used to identify the “don’t care” symbol, that is a symbol that matches without error with any symbol in Σ, and the expression X(d) denotes a string composed by d symbols X. Now, define a structured pattern a pattern of the form p = w1[k1 ] X(d1 ) w2[k2 ] . . . X(dr−1 ) wr[kr ] denoting a string composed by r words such that the i-th word has length ki and di don’t care symbols separate wi from wi+1 (1 ≤ i < r). An occurrence of the pattern p in SC is a substring s of a string in SC having the same length as p and such that each word wi of p is equal to the corresponding word swi of s. Analogously, an e-occurrence of the pattern p in SC is a substring e-s of SC having the same length as p and such that each 1
The Hamming distance between two words m and m is given by the minimum number of symbol substitutions to be performed on m in order to transform m into m.
Discovering Frequent Structured Patterns from String Databases
37
word wi of p has an Hamming distance at most e from the corresponding word e-swi of e-s. A structured pattern p is, therefore, a structured exact motif if there are at least q strings in SC containing an occurrence of p. Analogously, a pattern p is a structured e-motif if there are at least q strings in SC containing an e-occurrence of p. Obviously a structured exact motif is also a structured e-motif but not vice versa. In this paper we will address the following problems. Given a collection SC of strings and a quorum q: Problem 1. Find all structured exact motifs of the form m = w1[k1 ] X(d) w2[k2 ] . Problem 2. Find all structured exact motifs of the form m = w1[k1 ] X(d1 ) w2[k2 ] X(d2 ) . . . X(d(r−1) ) wr[kr ] , for r > 2. Moreover, fixed a maximum allowed error e: Problem 3. Find all structured e-motifs of the form e-m= w1[k1 ] X(d) w2[k2 ] . Problem 4. Find all structured e-motifs of the form e-m= w1[k1 ] X(d1 ) w2[k2 ] X(d2 ) . . . X(d(r−1) ) wr[kr ] , for r > 2.
3 3.1
Solving Problem 1 Overview
The algorithm we are going to define for solving Problem 1 exploits Subword Trees [1,2,5,10,11] for storing subwords of the strings in the collection SC possibly containing a structured motif. The basic idea underlying our approach is that of representing the structured patterns, candidate to be structured motifs, using links between subwords laying at distance d in SC. Each pair of subwords laying at distance d are suitably inserted in the Subword Tree and are identified by references to the Tree nodes representing them (node pointers). Links between subwords are expressed as pairs of references. We call these links d-links because they represent links between subwords laying at distance d. As an example, if Subword Tree nodes v1 and v2 associated, respectively, to the words w1 and w2 laying at distance d in a certain string of SC, the d-link (v1 , v2 ) is used to represent the pattern w1 X(d) w2 . Each d-link is associated with a number of occurrences, stating the number of strings in SC containing the pattern it represents. It is known that Subword Trees in compact form, representing the set of subwords in SC, can be built in linear time and occupies linear space in the size of SC. Differently from other approaches presented in the literature [6,14] the main idea of our approach is that of extracting the motifs during the construction of the Subword Tree. In particular, during the construction of the Subword Tree, we obtain the d-links representing the patterns, and compute the associated number of occurrences. The result yielded by our algorithm is the set of d-links having a number of occurrences greater than the quorum q along with the Subword Tree referred by d-links.
38
L. Palopoli and G. Terracina
One important issue to be dealt with in our approach is how and where to represent d-links. Indeed, storing d-links directly into the nodes of the Subword Tree T constructed from SC may lead to non-linear algorithms, which is a situation we wish to avoid. Therefore, we store and manage d-links into a separate support structure, say d-Tree, handled in its turn as a Subword Tree. Moreover, in order to properly exploit the features of Subword Trees, we represent the pointers of a d-link as strings (of 0s and 1s) and a d-link is denoted by the juxtaposition of such strings. For each pattern pi , the number of occurrences associated to the corresponding d-link li is stored into the node vleafi , representing li , and belonging to d-Tree. Note that vleafi is always a leaf node of d-Tree. Since we need to count the number of distinct strings of SC in which pi occur, a further information, indicating the last string in which pi has been found, must be stored into the node. The algorithm for solving Problem 1 works as follows. It scans the symbols in SC one at a time and, starting from the position of each of them, it considers 2 subwords w1 and w2 such that w2 lay at distance d from w1 . These subwords are inserted into the (main) Subword Tree T and, from the pointers to the nodes associated to these subwords, the algorithm creates a d-link l stating that w1 and w2 have been found at distance d. The algorithm then inserts l into the (support) Subword Tree d-Tree storing all the d-links extracted from SC. If the d-link l is a newly inserted one its number of occurrences is set to 1, otherwise it is updated only if it is the first time that the pattern is found in the current string. When the number of occurrences associated to a d-link equals the quorum q, the pattern identified by that d-link is recognized as a structured exact motif and, therefore, it is added to the set of results. 3.2
Algorithm P1
In this section we present our algorithm for solving Problem 1. The algorithm receives in input a collection SC of N strings, the lengths k1 and k2 of the subwords w1 and w2 , resp., the distance d and the quorum q and returns the set of structured exact motifs w1[k1 ] X(d) w2[k2 ] in SC. Algorithm P1 for solving Problem 1 Input: a collection SC of sequences, and four integers k1 , k2 , d and q representing, respectively the length of the first subword, the length of the second subword, the distance and the quorum. Output: The set Results of d-links representing structured exact motifs solving Problem 1 and the Subword Tree T obtained from SC. Type StringSet: Set of Strings; var T, d-Tree: Subword Tree; pf irst , psecond , pleaf : pointer; str, i, occ: integer; d-link, w1 , w2 : string;
Discovering Frequent Structured Patterns from String Databases
39
Results: StringSet; begin for str:=1 to NumberOfStrings(SC) do begin for i:=1 to StringLenght(SC,str) – (k1 +d+k2 ) do begin w1 :=Subword(SC,str,i,k1 ); w2 :=Subword(SC,str,i + k1 + d,k2 ); pf irst :=InsertString(T,w1 ); psecond :=InsertString(T,w2 ); d-link:=Convert(pf irst ,psecond ); pleaf :=InsertString(d-Tree,d-link); if (LastString(pleaf ) = str) then begin occ:=IncrementOccurrences(pleaf ); LastStringUpdate(pleaf ,str); if (occ = q) then AddResult(Results,d-link); end end; end; end
Procedures and functions used in the algorithm above are as follows. Function NumberOfStrings receives a collection SC of strings and returns the number of strings it contains. Function StringLength receives in input a collection SC of strings and a string index str; it yields in output the length of the string indexed by str. Function Subword receives a collection SC of strings and three integers str, i, k and yields the subword of length k starting from position i of the string indexed by str in SC. Function InsertString receives a Subword Tree T and a subword w; it inserts w in T and returns the pointer to the node of T associated to the last symbol of w. The function Convert takes in input two pointers pf irst and psecond ; it converts each pointer from its numeric to its string representation and juxtaposes the two obtained strings to obtain a single string representing a d-link. Function LastString receives a pointer pleaf to a leaf node of d-Tree and returns the index (stored therein) of the last string that caused updating of the associated number of occurrences (note that both the number of occurrences and the “last string” index are initialized to 0 when the node is created). Function IncrementOccurrences receives a pointer pleaf and increments the number of occurrences stored in the node pointed by pleaf ; the function returns the updated number of occurrences. The procedure LastStringUpdate takes in input a pointer pleaf and a string index str and updates the information stored in the node pointed by pleaf relative to the last string index that caused updating the associated number of occurrences. Finally, the procedure AddResult receives a set of strings Results and a string representing a d-link and adds it to the set Results. 3.3
Complexity Issues
In this section we analyze the complexity of Algorithm P1. Let n = |SC| denote the size (number of symbols) of the string collection SC, our algorithm extracts
40
L. Palopoli and G. Terracina
all structured exact motifs in just one scan of SC. In particular, it extracts the motifs during the construction of d-Tree. It has been shown that the construction of subword trees in compact form [1,11,2] has linear time complexity in n; therefore, the construction of T is O(n). Since we insert exactly one d-link for each subword extracted from SC, the construction of d-Tree is done in O(n) too. The information stored into each dTree leaf node, relative to the number of occurrences of that d-link and the last string that caused an update, can be checked/updated in constant time since we always have at disposal the pointer to that node. Thus, the overall time complexity of Algorithm P1 is O(n). Moreover, it requires O(n) additional space for storing the subword trees.
4 4.1
Solving Problem 2 Overview
Problem 2 can be solved in a quite straightforward way by slightly modifying Algorithm P1. The main difference regards the representation and creation of the d-links. As we have done in Algorithm P1, the algorithm for solving Problem 2 extracts all structured exact motifs during the construction of both T and d-Tree in a single scan of SC. In order to handle structured motifs of the form m = w1[k1 ] X(d1 )w2[k2 ] X(d2 ) . . . X(d(r−1) ) wr[kr ] we have to deal with r > 2 subwords at a time. Thus, a d-link representing a pattern of this kind, must be a cross link between r nodes of the subword tree T constructed from the strings in SC. d-links introduced in the previous section can be extended in a natural way to represent this kind of patterns by juxtaposing the r pointers to Subword Tree nodes associated to the r subwords to represent. The manipulation of these kind of d-links is performed in the same way as in the previous section; in particular, d-links are stored into a subword tree d-Tree, each d-link li is associated with information stating the number of occurrences in SC of the pattern represented by li and the index of the last string that caused updating this number of occurrences. Both these numbers are stored into the leaf node of d-Tree associated with li . Thus, the algorithm for solving Problem 2 works as follows. For each symbol s of SC, it extracts, starting from s, r subwords such that the i-th subword is at distance di from the (i+1)-th subword in SC; it inserts these subwords into the Subword Tree T . From the pointers to the nodes associated to these subwords it creates a d-link l and inserts l into the subword tree d-tree storing all the d-links extracted from SC. The updating of the number of occurrences of l is carried out in the same way as in Algorithm P1, in order to take into account occurrences of the pattern represented by l in distinct strings of SC. 4.2
Complexity Issues
As for the complexity analysis the following considerations can be drawn. Given a collection of strings SC, such that |SC| = n, which exact structured pattern of
Discovering Frequent Structured Patterns from String Databases
41
the form w1[k1 ] X(d1 ) w2[k2 ] X(d2 ) . . . X(d(r−1) ) wr[kr ] must be extracted from, the number of necessary subword insertions into the subword tree T is O(r ∗ n). The number of d-links generated by the algorithm is O(n), therefore, the overall complexity of the algorithm is O(r ∗ n).
5 5.1
Solving Problem 3 Overview
Solving Problem 3 is quite a more difficult task w.r.t. Problems 1 and 2. This because the difficulties arising from the structure of the motifs are coupled with the difficulties arising when the exact repetition requirement is relaxed. The former difficulty (i.e., motif structure) is dealt with the support of d-links as shown in the previous sections. The latter one requires a more involved analysis. Recall that, given a collection of strings SC, a quorum q and a number of errors e, a structured e-motif e-m is a structured pattern having at least q eoccurrences in SC. An e-occurrence of e-m in SC is a pattern p having the same structure as e-m and such that each of the subwords included in e-m have an Hamming distance at most equal to e from the corresponding subword of p. In this section we propose a technique for extracting structured e-motifs which exploits the concept of e-neighbor, defined next, for boosting the motif extraction process. Given a word w of length k, an alphabet Σ and a maximum allowed number e of mismatches e, there are Σi=1 (ki )(|Σ|−1)i different possible words at Hamming distance less than or equal to e from w (i.e., being e-ocurrences of w). We want to represent this set of words in a as compact as possible way. First these words can be grouped according to the positions in which mismatches with w occur; the e number of these groups is Σi=1 (ki ). By substituting the mismatching symbols with the “don’t care” symbol X, each group can be represented by a single word. Words of a group can be obtained by “instantiating” the Xs with specific symbols in the alphabet. Finally, note that because of the meaning associated to the “don’t care” symbols X, all strings containing l ≤ e X symbols are all represented by the (ke ) strings containing exactly e Xs. So, we can focus on those e−1 k (ke ) strings, disregarding the remaining Σi=1 (i ) ones. We call the set of strings that are obtained from w by substituting exactly e symbols of w by “don’t care” symbols, the e-neighbor Set of w; each word included in this set is called an e-neighbor of w. Note that the e-neighbor Set of a word w represents the whole set of e-occurrences of w. As an example, consider the word w = AGCT and an error e = 2. The e-neighbor Set of w is: {AGXX, AXCX, AXXT, XGCX, XGXT, XXCT }. It is worth observing that if the intersection between the e-neighbor Sets of two words w1 and w2 is not empty then w1 is an e-occurrence of w2 and vice versa, but the sets of e-occurrences of w1 and w2 might be different. Our solution to Problem 3 consists in two steps: – determining the occurrences of structured patterns formed by the e-neighbors of the subwords of SC lying at distance d; we will call this patterns e-neighbor
42
L. Palopoli and G. Terracina
patterns. Exact repetitions of e-neighbors are considered in the computation of the occurrences. – determining the e-occurrences of each structured pattern pi (candidate to be a structured e-motif); this is done by combining the number of occurrences of the e-neighbor patterns generated by pi . Both structured patterns and e-neighbor patterns are handled and stored, as described in the previous sections, with the support of Subword Trees. However, in order to combine the occurrences of two different e-neighbor patterns for determining the e-occurrences of the corresponding structured pattern, it is not sufficient to manipulate just the number of their occurrences. Indeed, the sets of strings in which two e-neighbor patterns have been found may overlap. Therefore, it is necessary to store, for each e-neighbor pattern, a boolean array of size equal to the number N of strings in the collection SC. This boolean array indicates the strings in the collection containing the corresponding e-neighbor pattern. We exploit an Extended Subword Tree for storing d-links associated to e-neighbor patterns. This is a standard Subword Tree storing into leaf nodes boolean arrays in the place of simple integers. 5.2
Algorithm P3 Algorithm P3 for solving Problem 3 Input: a collection SC of N sequences, and five integers k1 , k2 , d, q and e representing, respectively the length of the first subword, the length of the second subword, the distance, the quorum and the maximum number of errors allowed for each subword in the pattern; Output: The set Results of d-links representing structured e-motifs solving Problem 3 and the Subword Tree T obtained from SC. Type StringSet: Set of strings; var str,i : integer; w1 , w2 , ewi , ewj , d-link, e-d-link : string; Results, e-neighborSet1, e-neighborSet2 : StringSet; pf irst , psecond , pleaf :pointer; e-T, T, d-Tree : SubwordTree; e-d-Tree : ExtendedSubwordTree; presence : Array of N bits; begin for str:=1 to NumberOfStrings(SC) do begin for i:=1 to StringLenght(SC,str) – (k1 + d + k2 ) do begin w1 :=Subword(SC, str, i, k1 ); w2 :=Subword(SC,str, i + k1 + d, k2 ); e-neighborSet1:=Extract e-neighbors(w1 , e); e-neighborSet2:=Extract e-neighbors(w2 , e); for each ewi ∈ e-neighborSet1 do begin pf irst :=InsertString(e-T, ewi ); for each ewj ∈ e-neighborSet2 do begin psecond :=InsertString(e-T, ewj );
Discovering Frequent Structured Patterns from String Databases
43
e-d-link:=Convert(pf irst , psecond ); pleaf :=InsertString(e-d-Tree, e-d-link); SetPresence(pleaf , str); end; end; pf irst :=InsertString(T, w1 ); psecond :=InsertString(T, w2 ); d-link:=Convert(pf irst , psecond ); pleaf :=InsertString(d-Tree, d-link); end; end; for each d-link in d-Tree do begin RetrieveSubwords(d-link, d-Tree, w1 , w2 ); e-neighborSet1:=Extract e-neighbors(w1 ,e); e-neighborSet2:=Extract e-neighbors(w2 ,e); InitializePresenceArray(presence); for each ewi ∈ e-neighborSet1 do begin pf irst :=FindSubword(e-T, ewi ); for each ewj ∈ e-neighborSet2 do begin psecond :=FindSubword(e-T, ewj ); e-d-link:=Convert(pf irst ,psecond ); pleaf :=FindSubword(e-d-Tree, e-d-link); presence:=BitwiseOR(presence, GetPresence(pleaf )); end; end; if (NumberOfOccurrences(presence)> q) then AddResult(Results, d-link); end; end.
Here, functions NumberOfStrings, StringLength, Subword, InsertString, Convert and procedure AddResults are as described in Section 3. Function Extract e-neighbors receives a word w and an integer e and extracts the e-neighbor Set of w. Procedure SetPresence receives a pointer pleaf and a string index str; it sets to true the element of index str of the boolean array stored in the node referred by pleaf . Procedure RetrieveSubwords takes in input a d-link and the corresponding Subword Tree d-Tree and retrieves from d-Tree the words referred by the pointers composing the d-link. The procedure InitializePresenceArray simply initialize to false all the elements of the boolean array received in input. Function FindSubword receives a Subword Tree T and a word w and returns the pointer to the node of T associated to w. Function GetPresence takes in input a pointer p and yields the boolean array associated to the node referenced by p. Function BitwiseOR performs the OR operator between the elements of the two boolean arrays received in input. Finally, the function NumberOfOccurrences counts the number of true values contained in the boolean array received in input.
44
5.3
L. Palopoli and G. Terracina
Complexity Issues
Given a collection SC of N strings of average length equal to l so that lN = |SC|, in this section we analyze the complexity of Algorithm P3 for extracting the set of structured e-motifs of the form w1[k1 ] X(d) w2[k2 ] . Retrieving a subword from SC takes constant time. Extracting the e-neighbor Set of a word w of length k takes O(ν(k, e)) time, where ν(k, e) = (ke ), and the number of e-neighbors returned is O(ν(k, e)). The function SetPresence can be executed in constant time. Thus, the first step of the algorithm requires O(lN ν(k1 , e)ν(k2 , e)) As we have seen in Section 3.3, the maximum number of d-links in d-Tree is lN . Retrieving a pair of subwords from a d-link can be done in constant time since each d-link provides the pointers to the nodes in T associated to the subword pair. Initializing and performing the bitwise OR of boolean arrays can be done in time O(N ). Thus, the second step of the algorithm takes O(lN 2 ν(k1 , e)ν(k2 , e)). The overall time complexity of the algorithm is O(lN 2 ν(k1 , e)ν(k2 , e)). Finally, the algorithm requires O(lN 2 ν(k1 , e)ν(k2 , e)) space.
6
Solving Problem 4
Extending Algorithm P3 for solving Problem 4 is quite straightforward. It is necessary to extend d-links to represent r > 2 pointers as we have described in Section 4. Moreover, the construction of e-d-Tree must be modified in such a way to consider the combinations of r e-neighbors at a time. In particular, given r words from SC such that the word wi lay at distance di from the word w(i+1) , it is necessary to: (i) derive the r corresponding e-neighbor Sets; (ii) combine all the e-neighbors belonging to the r e-neighbor Sets; (iii) store the corresponding d-links into e-d-Tree. After this tasks are carried out, determining the e-occurrences of each structured pattern pi is performed in a way similar as that shown for Algorithm P 3. Given a collection SC of N strings of average length l so that lN = |SC|, the complexity of Algorithm P4 for extracting the set of structured e-motifs of the form w1[k1 ] X(d1 ) w2[k2 ] X(d2 ) . . . X(d(r−1) ) wr[kr ] is O(lN 2 × ν(k1 , e) × . . . × ν(kr , e)).
7
Discussion
In this section we compare our approach with some other techniques recently appeared in the literature. In [12] the problem of finding frequent structured patterns (called dyads in the paper) is demonstrated to be biologically meaningful by a real case study. In this paper, frequent patterns are extracted by enumerating all possible patterns over the alphabet having a particular structure and counting the number of occurrences of those patterns in the database. In [6] a wide range of patterns is considered. Particulary interesting, the authors propose a technique for extracting frequent patterns with very complex structures by exploiting pattern trees for representing the pattern space. While
Discovering Frequent Structured Patterns from String Databases
45
allowing to extract patterns with more complex structures than those considered in this paper, these algorithms are worst-case exponential in the length of patterns. The approach most closely related to our own is that presented in [14]. In this paper Suffix Trees, obtained from the input string set, are exploited to derive valid models which correspond to structured e-motifs defined in Section 2. In particular, models are incrementally obtained by recursively traversing (in a depth-first fashion) the (virtual) lexicographic trie M of all possible models of a fixed length k and the (actually built) suffix tree T of the sequences. Two algorithms are presented which both jump in the suffix tree to skip the d symbols separating the conserved regions of the patterns and exploit boolean arrays to combine the set of tuples containing the considered patterns. The approach proposed in [14] does not explicitly consider the case of exact repetitions of the patterns in the set, but only approximate ones. However, the exact case can be easily simulated by allowing e = 0 errors in the match between strings and patterns. In order to obtain structured e-motifs of the form w1[k1 ] X(d) w2[k2 ] , that is solving Problem 3 defined in Section 2, the cleverest algorithm proposed in [14] requires, O(N nk N 2 (e, k) + N n2k+d N (e, k)), where N is the number of strings in the collection, k = max(k1 , k2 ), nk ≤ lN indicates the number of Suffix Tree e nodes at level k, N (e, k) = Σi=0 (ki )(|Σ|−1)i ≤ k e |Σ|e is the number of strings at an Hamming distance e from a model and N nk N (e, k) is the number of possible models of length k that can be generated. In case of exact match (i.e., solving Problem 1), the complexity above reduces to O(N nk +N n2k+d ). These measures do not take into account the construction of the Suffix Tree from the strings. Recall that the complexity of our Algorithm P1 for solving Problem 1 (exact repetitions) is O(lN ), where lN = |SC|, N is the number of strings in the collection and l is the average length of such strings, whereas the complexity of our Algorithm P3 for solving Problem 3 is O(lN 2 ν(k1 , e)ν(k2 , e)), where ν(k, e) = (ke ). Thus, the main improvements of our approach w.r.t. [14] are the following: (i) Our approach solves Problem 1 during the construction of support structures; in particular, in Algorithm 1, all structured exact motifs are discovered during the construction of the Subword Tree; conversely the approach of [14] requires the availability of the Suffix Tree. (ii) The complexity of our approach for solving Problem 3 does not depend on the dimension of the alphabet (this is significant when dealing with large alphabets); indeed, the term ν(k, e) above depends only on k and e, whereas in [14] the term N (e, k) is bounded by k e |Σ|e . We are able to achieve this result by exploiting “don’t care” symbols and e-neighbors. (iii) Our algorithms do not depend on the number of symbols X(d) separating the words w1[k1 ] and w2[k2 ] in the patterns; this improvement is a direct result of the exploitation of d-links. Indeed, d-links allow us to retrieve pairs of words in the Subword Tree laying at distance d simply by analyzing the words stored in the tree. On the contrary, in [14], a word w is related to the other ones laying at distance d by jumping down in the Suffix Tree of d levels, which makes this algorithm complexity dependent form d. Moreover, this imposes the exploitation of boolean presence arrays even for solving Problem 1.
46
L. Palopoli and G. Terracina
References 1. A. Amir, M. Farach, Z. Galil, R. Giancarlo, and K. Park. Dinamic dictionary matching. Journal of Computer and System Science, 49:208–222, 1994. 2. A. Apostolico and M. Crochemore. String matching for a deluge survival kit. Handbook of Massive Data Sets, To appear. 3. A. Bairoch. PROSITE: A dictionary of protein sites and patterns. Nucleic Acid Research, 20:2013–2018, 1992. 4. G. Benson. An algorithm for finding tandem repeats of unspecified pattern size. In Proceedings of ACM Recomb, pages 20–29, 1998. 5. P. Bieganski, J. Riedl, J.V. Carlis, and E.M. Retzel. Generalized suffix trees for biological sequence data: Applications and implementations. In Proc. of the 27th Hawai Int. Conf. on Systems Science, pages 35–44. IEEE Computer Society Press, 1994. 6. A. Brazma, I. Jonassen, I. Eidhammer, and D. Gilbert. Approaches to the automatic discovery of patterns in biosequences. Journal of Computational Biology, 5(2):277–304, 1998. 7. Y.M. Fraenkel, Y. Mandel, D. Friedberg, and H. Margalit. Identification of common motifs in unaligned dna sequnces: application to escherichia coli lrp regulon. Computer Applied Bioscience, 11:379–387, 1995. 8. D.J. Galas, M. Eggert, and M.S. Waterman. Rigorous pattern-recognition methods for dna sequences: Analysis of promoter sequences from escherichia coli. J. of Molecular Biology, 186:117–128, 1985. 9. C.A. Gross, M. Lonetto, and R. Losick. Bacterial sigma factors. Transcriptional Regulation, 1:129–176, 1992. 10. D. Gusfield. Algorithms on Strings, Trees and Sequences: Computer Science and Computational Biology. Cambrige University Press, 1997. 11. D. Gusfield, G. M. Landau, and B. Schieber. An efficient algorithm of all pairs suffix-prefix problem. Information Processing Letters, 41:181–185, 1992. 12. J. Helden, A.F. Rios, and J. Collado-Vides. Discovering regulatory elements in noncoding sequences by analysis of spaced dyads. Nucleic Acids Research, 28(8):1808– 1818, 2000. 13. A. Klingenhofen, K. Frech, K. Quandt, and T. Werner. Functional promoter modules can be detected by formal methods independent of overall sequence similarity. Bioinformatics, 15:180–186, 1999. 14. L. Marsan and M.F. Sagot. Algorithms for extracting structured motifs using a suffix tree with application to promoter and regulatory site consensus identification. Journal of Computational Biology, 7:345–360, 2000. 15. M.F. Sagot and E.W. Myers. Identifying satellites in nucleic acid sequences. In Proc. of ACM RECOMB, pages 234–242, 1998. 16. H.O. Smith, T.M. Annau, and S. Chandrasegaran. Finding sequence motifs in groups of functionally related proteins. In Proc. of National Academy of Science, pages 118–122, U.S.A., 1990. 17. R.L. Tatusov, S.F. Altschul, and E.V. Koonin. Detection of conserved segments in proteins. In Proc. of National Academy of Science, pages 12091–12095, U.S.A., 1994.
Discovery in Hydrating Plaster Using Machine Learning Methods Judith E. Devaney and John G. Hagedorn National Institute of Standards and Technology, Gaithersburg MD, 20899-8951, USA, {judith.devaney,john.hagedorn}@nist.gov, http://math.nist.gov/mcsd/savg
Abstract. We apply multiple machine learning methods to obtain concise rules that are highly predictive of scientifically meaningful classes in hydrating plaster over multiple time periods. We use three dimensional data obtained through X-ray microtomography at greater than one micron resolution per voxel at five times in the hydration process: powder, after 4 hours, 7 hours, 15.5 hours, and after 6 days of hydration. Using statistics based on locality, we create vectors containing eight attributes for subsets of size 1003 of the data and use the autoclass unsupervised classification system to label the attribute vectors into three separate classes. Following this, we use the C5 decision tree software to separate the three classes into two parts: class 0 and 1, and class 0 and 2. We use our locally developed procedural genetic programming system, GPP, to create simple rules for these. The resulting collection of simple rules are tested on a separate 1003 subset of the plaster datasets that had been labeled with their autoclass predictions. The rules were found to have both high sensitivity and high positive predictive value. The classes accurately identify important structural comonents in the hydrating plaster. Morover, the rules identify the center of the local distribution as a critical factor in separating the classes.
1
Introduction
Plaster of paris is a widely used material of economic importance [1]. For example, the porcelain industry maintains large numbers of molds whose strength, durability, and ability to absorb water impact the industry’s costs [2]. Plaster powder is formed by calcining gypsum (calcium sulfate dihydrate, CaSO4 · 2H2 O) to form calcium sulfate hemihydrate (CaSO4 · 12 H2 O). The solid plaster is then formed by adding water (hydration) to the powder and allowing the mixture to set. The equations are [1]: Calcination: CaSO4 · 2H2 O = CaSO4 · 12 H2 O + 32 H2 O Hydration: CaSO4 · 12 H2 O + 32 H2 O = CaSO4 · 2H2 O During hydration, an interlocking network of gypsum crystals forms. See Figure 1 for a scanning electron micrograph (900X) [3] of precipitated gypsum crystals (CaSO4 · 2H2 O). This crystalline network is the foundation of the strength, S. Lange, K. Satoh, and C.H. Smith (Eds.): DS 2002, LNCS 2534, pp. 47–58, 2002. c Springer-Verlag Berlin Heidelberg 2002
48
J.E. Devaney and J.G. Hagedorn
durability, and absorptivity of the plaster [1]. However, the final properties of the set plaster are dependent on many things such as the water-solid ratio in hydration, impurities in the plaster, additives, temperature, and production conditions [3]. Moreover, these interact. For example, as the water-solid ratio in the hydrating plaster increases, the volume fraction of porosity increases, absorptivity of the plaster increases, but the strength and durability decrease [1]. There is much to learn about plaster; even the form of the kinetic equations (fraction of plaster reacted versus time) is not agreed upon [4][5][6]. Understanding the process of setting plaster as well as being able to predict its final properties is of scientific as well as economic interest.
Fig. 1. A scanning electron micrograph (900X) [3] of precipitated gypsum crystals (CaSO4 · 2H2 O).
2
High Resolution Data
Recently, X-ray microtomography has been used to obtain the unprecedented resolution of 0.95µm per voxel in three dimensional images of hydrating plaster. Commercial grade plaster of paris was mixed with a water-to-solids mass ratio of 1.0 and viewed with X-ray microtomography after 4, 7, 15.5 hours and 6 days. Additionally, a separate sample of plaster powder was imaged. This resulted in five images of plaster of size 10243 . This is gray scale data with each voxel varying from 0 to 255 [7][8].
3
Methodology
We seek simple rules to describe and predict structural components in hydrating plaster such as hydration products (gypsum crystals), unhydrated plaster, and porosity (holes). With materials like cement, it is straightforward to obtain such components through thresholding of the brightness histogram. However, this is
Discovery in Hydrating Plaster Using Machine Learning Methods
49
not the case with plaster [8]. Since the problem is not straightforward, we use multiple methods from machine learning to obtain the rules. We use a combination of unsupervised classification, decision trees, and genetic programming to obtain these rules. The rules are developed on a 1003 subset of the data taken at the same place in each dataset. The rules are tested on a completely separate 1003 subset of the data taken at the same place in each dataset. 3.1
Unsupervised Classification
Since the data is unlabeled, we start the discovery process with an unsupervised classifier. We use autoclass [9][10][11][12][13] which has been used successfully for discovery. In the creation of attributes for input to autoclass, we follow the Principle of locality [14], wherein natural laws are viewed as the consequence of small-scale regularities. Since the particle sizes may be as small as a few microns [8], we choose as our scale a 33 cube centered on each pixel in the image. Using simple statistics on these small cubes, we create eight attributes for each pixel as input vectors to autoclass, as described in the following table. Name A0 A1 A2 A3 A4 A5 A6 A7
Definition gray level value of pixel itself neighborhood midrange neighborhood variance about midrange neighborhood range neighborhood minimum neighborhood maximum neighborhood median neighborhood mean
Hence, for the 1003 training subcube, this results in 1, 000, 000 vectors. Because materials scientists are interested in three classes within hydrating plaster [15], we constrain autoclass to seek three classes. Classification runs were performed for each training subcube for powder, 4 hours, 7 hours, 15.5 hours, and 6 days. Since this data is at a new resolution, we do not have pre-labeled data to compare it with, or experts who can label it. We validate the classification through a visual comparison with the classes obtained. The 1003 dataset used for training is small enough to look at all one hundred 1002 images in the dataset and in the classification. A hundred 1002 images can be printed on an 8 12 by 11 page in a ten by ten array making comparisons straightforward. Due to space considerations, we reproduce three images from each array here. Each image is taken at z = 0, 30, 60 in the image array. Figure 2 shows the data and classes for the plaster powder. The data is on the left, with z = 0 at the bottom, z = 30 in the middle, and z = 60 at the top. The corresponding classification for each plane is to the right of the data. Class 0 is black, class 1 is gray, and class 2 is white.
50
J.E. Devaney and J.G. Hagedorn
Fig. 2. Comparison of three 1002 slices of original data with corresponding classification for powder. The data is in the left column, with z = 0 at the bottom, z = 30 in the middle, and z = 60 at the top. The corresponding classification for each plane is to the right of the data.
It is immediately obvious that autoclass has picked up the basic structure of the data. The plaster particles are in class 1. Figure 3 shows the equivalent images for 4, 7, 15,5 hours and 6 days of hydration. Again the structure in the data matches the structure of the classification. In the classified data, Class 2 (the white area) is the pore space, Class 1 (the grey area) identifies the crystalline network and the unhydrated plaster, and Class 0 (the black area) represents the boundary region. Figure 4 shows this difference clearly in renderings of the individual classes in their three dimensional configuration. Plaster hardens with little change in external volume, but since the volume of the hydration products is smaller than the original material plus water, voids occur inside [16]. In our classification, the space seems to form around the gypsum crystals. 3.2
Decision Tree
In order to gain better insight into the classifications for each time period, we seek comprehensible representations of the classification algorithms. That is, we wanted to find relatively simple rules to determine each element’s class based on its eight attributes. But autoclass operates in a “black-box” fashion. The algorithm by which it classifies and predicts elements is opaque to the user.
Discovery in Hydrating Plaster Using Machine Learning Methods
51
Fig. 3. Comparisons of original data with classification for 4 hours (top left), 7 hours (top right), 15.5 hours (bottom left), and 6 days (bottom right). For each time, three 1002 slices of original data are in the left column, with z = 0 at the bottom, z = 30 in the middle, and z = 60 at the top. The corresponding classification for each plane is to the right of the data.
52
J.E. Devaney and J.G. Hagedorn
To derive more transparent statements of the classification schemes, we used a decision tree, C5 [17]1 . C5 is the commercial successor to C4.5 [18], which has been used extensively for learning. Runs of C5 on the autoclass labeled attributes produced incomprehensible trees with thousands of nodes. However, the component classes in the brightness histograms in Figure 5 indicated that class 2 and class 1 were easily separable. Ten fold cross validation on the combined class 1 and 2 showed that this was the case in four of the five datasets. Three of the datasets yielded single node decision stumps with less than five missclassifications over hundreds of thousands of cases for powder, 4 hours, and 15.5 hours. All of these branched on attribute A1. The fourth simple classification was for the 7 hour dataset. This also yielded a single node decision stump; however, this branched on A7. For uniformity in the final rules across the hydration times, the 7 hour case was rerun requiring C5 to use only A1 to get a single best split on this attribute. The six day dataset did not yield a simple decision tree for the combined class 1 and 2. So it was also run with A1 as the only attribute to get the best split for input into the next phase, which was genetic programming to obtain complete and simple rules. The attribute A1 is the local midrange. The midrange is a robust estimator of the center of a short tailed distribution [19]. Since the range is limited for each neighborhood to 0 − 255, this is the situation here. So all the rules are now of the form: if the center of the local distribution is ≤ x ....
Fig. 4. Three dimensional renderings of individual classes (class 0 is on the left, class1 is in the middle, and class 2 is on the right) at 4 hours of hydration.
3.3
Genetic Programming
Genetic Progamming [20][21][22] is a technique that derives a sequence of operations (chosen from a user defined set) that have high fitness (as defined by the user) using techniques inspired by evolution. We use GPP (Genetic Program1
The identification of any commercial product or trade name does not imply either endorsement or recommendation by the National Institute of Standards and Technology.
Discovery in Hydrating Plaster Using Machine Learning Methods
53
Fig. 5. Brightness histograms for whole datasets as well as component classes for powder, 4 hours, 7 hours, 15.5 hours, and 6 days.
ming - Procedural) [23][24], a procedural genetic programming system that we have developed. GPP was used to derive simple, understandable formulae that closely match the original classifications provided by autoclass. Before GPP is used, the class 1/2 decision algorithm is determined by C5 as described above. GPP is then used to derive the class 0/1 and the class 0/2 decision algorithms. The method for using GPP in this problem followed the following steps for each desired classification: – – – – – –
Prepare training data sets from the classified data sets. Select parameters, such as the operator set, for the GPP runs. Construct a fitness function to measure algorithm effectiveness. Execute a set of GPP runs. Select the run with results that most closely match the original classification. Simplify the GPP-produced algorithm to a succinct form.
All GPP runs were done with the same set of operating parameters. We used a small set of arithmetic and logical operators, a population size of 500, and the maximum number of generations for a single evolution was 50. The fitness function that we used is based on the correlation between the algorithm’s classifications with the actual classifications. The fact that this is a two-valued problem simpifies the calculation of the correlation. We use the formulation by Matthews[25], which has been used in the context of genetic programming by Koza[21]. The correlation is given by:
54
J.E. Devaney and J.G. Hagedorn
T p T n − Fn Fp (Tn + Fn )(Tn + Fp )(Tp + Fn )(Tp + Fp ) where: – – – –
Tp is the number of true positives Tn is the number of true negatives Fp is the number of false positives Fn is the number of false negatives
The correlation is evaluated based on the execution of an algorithm on the appropriate training set. This correlation value varies from -1 to 1, where 1 indicates perfect performance on the training set and -1 indicated a perfectly incorrect performance on the training set. Because these decision algorithms are binary in nature, we can turn a very bad algorithm into a very good algorithm simply by inverting each decision. In terms of the correlation value, this means that we can regard a decision algorithm with a correlation of -0.6 to be just as good as an algorithm with correlation +0.6. So our fitness function is the absolute value of the correlation value given above. Each GPP run seeks to evolve an algorithm that maximizes this fitness value. For each decision algorithm to be derived, five hundred GPP runs were made. Each run differed only in the seed to the random number generator. Given the resources available to us, we easily ran all 12 sets of runs over a single night. After completing a set of five hundred runs we evaluated the algorithm generated by each run on the full data sets. This enables us to select the best performing algorithm. We then use a symbolic computation system (Maple [26]) to reduce the algorithms to simpler forms.
4
Results
Relatively simple decision rules were derived for all five time steps and for each of the required classifications. After deriving the rules, we sought to evaluate their effectiveness relative to the original autoclass classification. 4.1
Rules
Here are the classification algorithms that were derived by C5 and GPP. Note that the derived rules are succinct and the entire classification algorithm for a particular time step is quite compact. Powder if A1 ≤ 42. then (class is either 2 or 0) if .518494A3 + .019318A6 = 2
Discovery in Hydrating Plaster Using Machine Learning Methods
then class = 0 else class = 2 else (class is either 1 or 0) if (7235./A7) ≤ (A7 + A4 + A1) then class = 1 else class = 0 4 hours if A1 ≤ 27. then (class is either 2 or 0) if A6 = 0 then if 0.5 + .06145A1 = 2 then class = 0 else class = 2 then if 0.5 + .06145A1 + .003373A3 = 2 then class = 0 else class = 2 else (class is either 1 or 0) if (6323.0/A1) < (A0 + A4 + 0.692129A6 + A7) then class = 1 else class = 0 7 hours if A1 ≤ 42.5 then (class is either 2 or 0) if .527467A1 + .027467A7 = 2 then class = 0 else class = 2 else (class is either 1 or 0) if 0.5 + 0.008515A7 = 1 then class = 1 else class = 0 15.5 hours if A1 ≤ 30. then (class is either 2 or 0) if .525849(A6 + A3) = 2 then class = 0 else class = 2 else (class is either 1 or 0) if A0 = 0 then if A4 < (5321.0/A7) − (a0 + a7) then class = 0 else class = 1 else if A4 < (5321.0/A7) − a3
55
56
J.E. Devaney and J.G. Hagedorn
then class = 0 else class = 1 6 days if A1 ≤ 28. then class = 2 else (class is either 1 or 0) if A0 + (A1 − (4643/A7)) > 0 then class = 0 else class = 1 4.2
Evaluation of the Rules
We use sensitivity and positive predictive value [27] as metrics to evaluate our rules. A rule can be optimal with respect to a particular classification in two ways. The rule can be very successful at seeing a class when it is there. This is called its sensitivity. And the rule can be very successful at identifying the class in the presence of other classes. This is called its positive predictive value. Let Tp be the true positives. Let Fp be the false positives. Let Fn be the false negatives. Then: Sensitivity
=
Positive-Predictive-Value =
Tp Tp +Fn Tp Tp +Fp
In a confusion matrix, sensitivity is accuracy across a row; positive predictive value is accuracy down a column. We test our classification rules with a completely different 1003 subcube of data from each of the five time periods. To test the rules we first compute the same attribute vectors for the new dataset. Then we use the prediction capability of autoclass to label the vectors. Finally, we use the above rules to create confusion matrices of the preditions for each of the time periods. We derive the sensitivity and positive predictive values for class and time period. The derived rules are all highly predictive as shown in the following table. Dataset Powder 4 Hour 7 Hour 15.5 Hour 6 Day
5
Sensitivity Positive-Predictive-Value class 0 class 1 class 2 class 0 class 1 class 2 0.94 0.88 0.97 0.86 0.97 0.95 0.97 0.96 0.97 0.93 0.98 0.997 0.95 0.95 0.99 0.95 0.95 0.99 0.96 0.93 0.98 0.93 0.96 0.996 0.97 0.96 0.996 0.99 0.95 0.98
Conclusions and Future Work
We have taken unclassified data and created a set of simple rules that accurately predict the class of unseen data for plaster powder, and plaster after 4, 7, 15.5
Discovery in Hydrating Plaster Using Machine Learning Methods
57
hours, and 6 days of hydration. This was accomplished using a combination of three machine learning methods, providing results and insight that were not possible with any one of the techniques. Our work on plaster has just begun, however. First, we would like to develop a better method for validating the classifications. One approach is to generate simulated plaster data sets for which proper classifications are known, for example using computer model microstructures designed to mimic the Plaster of Paris system [28]. We will also be working with an expert to label manually small subsets of the X-ray tomography data sets. These labeled data will then be used for training and validation. This will likely result in refinement of our rules. Next we would like to develop equations that accurately predict the class regardless of the time of hydration, i.e. that work over the whole hydration period. We will need additional data to include variations with respect to the parameters that can influence the setting process and the resultant properties of plaster. Finally, we would like to predict physical characteristics of classes with equations, instead of predicting classes. Plaster is an interesting and exciting topic for automated discovery methods. We look forward to extending our study. Acknowledgements. We would like to thank Dale Bentz for his encouragement and support.
References 1. Kingrey, W.D., Bowen, H.K., Uhlmann, D.R.: Introduction to Ceramics. John Wiley and Sons, New York (1976) 2. Bullard, J.W.: Personal communication (2002) 3. Clifton, J.R.: Some aspects of the setting and hardening of gypsum plaster. Technical Note 755, NBS (1973) 4. Hand, R.J.: The kinetics of hydration of calcium sulphate hemihydrate: A critical comparison of the models in the literature. Cement and Concrete Research 24 (1994) 885–895 5. Ridge, M.J.: A discussion of the paper: The kinetics of hydration of calcium sulphate hemihydrate: A critical comparison of the models in the literature by r. j. hand. Cement and Concrete Research 25 (1995) 224 6. Hand, R.J.: A reply to a discussion by m. j. ridge of the paper: The kinetics of hydration of calcium sulphate hemihydrate: A critical comparison of the models in the literature. Cement and Concrete Research 25 (1995) 225–226 7. D. P. Bentz, S. Mizell, S. G. Satterfield, J. E. Devaney, W. L. George, P. M. Ketcham, J. Graham, J. Porterfield, D. Quenard, F. Vallee, H. Sallee, E. Boller, J. Baruchel: The Visible Cement Dataset. J. Res. Natl. Inst. Stand. Technol. 107 (2002) 137–148 8. The visible cement dataset (2002) [online] . 9. Cheeseman, R., Kelley, J., Self, M., Taylor, W., Freeman, D.: Autoclass: A bayesian classification system. In: Proceedings of the Fifth International Conference on Machine Learning, San Francisco, CA, Morgan Kaufman (1988) 65–74
58
J.E. Devaney and J.G. Hagedorn
10. Cheeseman, P.: On finding the most probable model. In Shrager, J., Langley, P., eds.: Computational Models of Discovery and Theory Formation. Morgan Kaufman, San Francisco, CA (1991) 73–96 11. Stutz, J., Cheeseman, P.: Bayesian classification (autoclass): Theory and results. In Fayyad, U.M., Piatetsky-Shapiro, G., Smyth, P., Uthurusamy, R., eds.: Advances in Knowledge Discovery and Data Mining. AAAI Press, Menlo Park, CA (1995) 12. Kanefsky, B., Stutz, J., Cheeseman, P., Taylor, W., Clifton, J.R.: An improved automatic classification of a landsat/tm image from kansas (fife). Technical Report FIA-94-01, NASA AMES (1994) 13. Goebel, J., Volk, K., Walker, H., Gerbault, P., Cheeseman, P., Self, M., Stutc, J., Taylor, W.: A bayesian classifiection of the iras lrs atlas. Astronomy and Astrophysics 222 (1989) L5–L8 14. Reichenbach, H.: Atom and Cosmos. Dover Publications, Inc., Mineola, New York (1932) (First published in 1930 as Atom und Kosmos.). 15. Bentz, D.P.: Personal communication (2002) 16. Sattler, H., Bruckner, H.P.: Changes in volume and density during the hydration of gypsum binders as a function of the quantity of water available. ZKG International 54 (2001) 522 17. : C5 (2002) [online] . 18. Quinlan, J.R.: C4.5: Programs for Machine Learning. Morgan Klauffann, San Mateo (1993) 19. Crow, E.L., Siddiqui, M.N.: Robust estimation of location. Journal of the American Statistical Associatiuon 63 (1967) 363–389 20. Koza, J.R.: Genetic Programming: On the Programming of Computers by Means of Natural Selection. MIT Press (1992) 21. Koza, J.R.: Genetic Programming II: Automatic Discovery of Reusable Programs. MIT Press (1994) 22. Koza, J.R., Andre, D., Bennett III, F.H., Keane, M.: Genetic Programming 3: Darwinian Invention and Problem Solving. Morgan Kaufman (1999) 23. Hagedorn, J., Devaney, J.: A genetic programming system with a procedural program representation. In: 2001 Genetic and Evolutionary Computation Conference Late Breaking Papers. (2001) 152–159 http://math.nist.gov/mcsd/savg/papers˜. 24. Devaney, J., Hagedorn, J., Nicolas, O., Garg, G., Samson, A., Michel, M.: A genetic programming ecosystem. In: Proceedings 15th International Parallel and Distributed Processing Symposium, IEEE Computer Society (2001) 131 http://math.nist.gov/mcsd/savg/papers˜. 25. Matthews, B.W.: Comparison of the predicted and observed secondary structure of t4 phage lysozyme. Biochimica et Biophysica Acta 405 (1975) 442–451 26. Monagan, M.B., Geddes, K.O., Heal, K.M., Labahn, G., Vorkoetter, S.M., McCarron, J.: Maple 6 Programming Guide. Waterloo Maple Inc., Waterloo, Ontario, Canada (2000) 27. Lathrop, R., Erbdster, T., Smith, R., Winston, P., Smith, T.: Integrating ai with sequence analysis. In Hunter, L., ed.: Aritifical Intelligence and Molecular Biology, Cambridge, MA (1993) 28. Meille, S., Garboczi, E.J.: Linear elastic properties of 2-d and 3-d modles of porous materials made from elongated objects. Mod. Sim. Mater. Sci 9 (2001) 1–20
Revising Qualitative Models of Gene Regulation Kazumi Saito1 , Stephen Bay2 , and Pat Langley2 1
NTT Communication Science Laboratories 2-4 Hikaridai, Seika, Soraku, Kyoto 619-0237 Japan [email protected] 2 Institute for the Study of Learning and Expertise 2164 Staunton Court, Palo Alto, CA 94306 USA [email protected], [email protected]
Abstract. We present an approach to revising qualitative causal models of gene regulation with DNA microarray data. The method combines search through a space of variable orderings with search through a space of parameters on causal links, with weight decay driving the model toward integer values. We illustrate the technique on a model of photosynthesis regulation and associated microarray data. Experiments with synthetic data that varied distance from the target model, noise, and number of training cases suggest the method is robust with respect to these factors. In closing, we suggest directions for future research and discuss related work on inducing causal regulatory models.
1
Introduction and Motivation
Like other sciences, biology requires that its models fit available data. However, as the field moves from a focus on isolated processes to system-level behaviors, developing and evaluating models has become increasingly difficult. This challenge has become especially clear with respect to models of gene regulation, which attempt to explain complex interactions in which the expression levels of some genes influence the expression levels of others. A related challenge concerns a shift in the nature of biological data collection from focused experiments, which involve only a few variables, to cDNA microarrays, which measure thousands of expression levels at the same time. In this paper, we describe an approach that takes advantage of such nonexperimental data to revise existing models of gene regulation. Our method uses these data, combined with knowledge about the domain, to direct search for a model that better explains the observations. We emphasize qualitative causal accounts because biologists typically cast their regulatory models in this form. We focus on model revision, rather than constructing models from scratch, because biologists often have partial models for the systems they study. We begin with a brief review of molecular biology and biochemistry, including the central notion of gene regulation, then present an existing regulatory model of photosynthesis. After this, we describe our method for using microarray data to improve such models, which combines ideas from learning in neural networks S. Lange, K. Satoh, and C.H. Smith (Eds.): DS 2002, LNCS 2534, pp. 59–70, 2002. c Springer-Verlag Berlin Heidelberg 2002
60
K. Saito, S. Bay, and P. Langley
and the notion of minimum description length. Next we report experimental studies of the method that draws on both biological and synthetic data, along with the results of these experiments. In closing, we suggest directions for future research and discuss related work on inducing causal models of gene regulation.
2
Qualitative Causal Models of Gene Regulation
A gene is a fundamental unit of heredity that determines an organism’s physical traits. It is an ordered sequence of nucleotides in deoxyribonucleic acid (DNA) located at at specific position on a chromosome. Genes encode functional products, called proteins, that determine the structure, function, and regulation of an organism’s cells and tissues. The gene’s nucleotide sequence is used to construct proteins through a multiple stage process. In brief, the enzyme RNA polymerase transcribes each gene into a complementary strand of messenger ribonucleic acid (mRNA) using the DNA as a template. Ribosomes then translate the mRNA into a specific sequence of amino acids forming a protein. Transcription is controlled through the RNA polymerase by transcription factors that let it target specific points on the DNA. The transcription factors may themselves be controlled through signalling cascades that relay signals from cellular or extra-cellular events. Typically, a signalling cascade phosphorylates (or dephosphorylates) a transcription factor, changing its conformation (i.e., physical structure) and its ability to bind to the transcription site. Translation is controlled by many different mechanisms, including repressors binding to mRNA that prevents translation into proteins. In our work, we focus on revising biological models that relate external cell signals to changes in gene transcription (as measured by mRNA) and, ultimately, phenotype. Specifically, we look at a model of photosynthesis regulation that is intended to explain why Cyanobacteria bleaches when exposed to high light conditions and how this protects the organism. This model, shown in Figure 1, was adapted from a model provided by a microbiologist (Grossman et al., 2001).1 Each node in the model corresponds to an observable or theoretical variable that denotes a measurable stimulus, gene expression level, or physical characteristic. Each link stands for a causal biological process through which one variable influences another. Solid lines in the figure denote internal processes, while dashes indicate processes connected to the environment. The model states that changes in light level modulate the expression of dspA, a protein hypothesized to serve as a sensor. This in turn regulates NBLR and NBLA expression, which then reduces the number of phycobilisome (PBS) rods that absorb light. The level of PBS is measured photometrically as the organism’s greenness. The reduction in PBS protects the organism’s health by reducing absorption of light, which can be damaging at high levels. The organism’s health 1
The paper describes an initial model for high light response in the Cyanobacterium S ynechococcus. This model was modified slightly for the Cyanobacterium used in our experiments, S ynechocystis PCC6803, by actions such as replacing nblS with its homolog dspA.
Revising Models of Gene Regulation NBLR
+
NBLA
−
PBS
+ Light
+
dspA
− + RR
− −
−
cpcB psbA2 psbA1
61
Health +
Photo
−
+
Fig. 1. Initial model for photosynthesis regulation of wild type Cyanobacteria.
under high light conditions can be measured in terms of the culture density. The sensor dspA impacts health through a second pathway by influencing an unknown response regulator RR, which in turn down regulates expression of the gene products psbA1, psbA2, and cpcB. The first two positively influence the level of photosynthetic activity (Photo) by altering the structure of the photosystem. If left unregulated, this second pathway would also damage the organism in high light conditions. Although the model incorporates quantitative variables, it is qualitative in that it specifies cause and effect but not the exact numerical form of the relationship. For example, one causal link indicates that increases in NBLR will increase NBLA, but it does not specify the form of the relationship, nor does it specify any parameters. The model is both partial and abstract. The biologist who proposed the model made no claim about its completeness and clearly viewed it as a working hypothesis to which additional genes and processes should be added as indicated by new data. Some links are abstract in the sense that they denote entire chains of subprocesses. For example, the link from dspA to NBLR stands for a signaling pathway, the details of which are not relevant at this level of analysis. The model also includes a theoretical variable RR, an unspecified gene (or possibly a set of genes) that acts as an intermediary controller.
3
An Approach to Revising Qualitative Causal Models
In this paper, we represent causal models in terms of linear relationships among variables. That is, each quantitative variable x(i) is represented with an equation of the form x(i) =
i−1
A(i, j)x(j) + b(i) ,
(1)
j=1
where A(i, j) describes the causal effect of variable x(j) on x(i) and b(i) is an additive constant. The variables in a model are ordered and variable x(i) can only be influenced by those variables that come before it in the causal ordering.
62
K. Saito, S. Bay, and P. Langley
Using matrix form, we can represent the equations for all x(i), i = 1..n, as x = Ax + b. In this formulation, A(i, j) = 0 if i ≤ j, where A(i, j) denotes the element in row i and column j of A. This constraint enforces the causal ordering on the variables. A model is completely specified by an ordering of variables in x and an assignment of values to all elements of A and b that satisfy the above constraints. This defines the space of models that our revision method will consider. However, we still need some way to map an initial biological model onto this notation. If we let A0 and b0 denote the initial model, then we can transform qualitative models like that in Figure 1 into a matrix A0 by setting A(i, j) = 1 if there is a positive link from variable j to i in the model, A(i, j) = −1 if the link is negative, and A(i, j) = 0 otherwise. We set the vector b0 to zero for all its elements. Given A0 , b0 , and observations on x, we learn new values for A and b by 1. Picking an initial ordering for variables in x; 2. Learning the best real-valued matrix A according to a score function that penalizes for differences from A0 , and is subject to the ordering constraints; 3. Swapping variables in the ordering and going to step 2 (i.e., performing hillclimbing search in the space of variable orderings), continuing until the score obtained no longer improves; and 4. Transforming the real matrix A that has the best score into a discrete version with A(i, j) ∈ −1, 0, 1 with a thresholding method. Step 1 in this revision algorithm determines the starting state of the search. Our approach selects a random ordering that is consistent with the partial ordering implied by the initial model. During step 2, our method invokes an approach to equation revision that transforms the equation x = Ax + b into a neural network, revises weights in that network, and then transforms the network back into equations in a fashion similar to that described by Saito et al. (2001). This neural network approach uses a minimum description length (Rissanen, 1989) criterion during training to penalize models that differ from the initial model. For example, suppose w0 is the parameter vector of the neural network that corresponds to the initial model. We define our revision task as finding a w that lets the network closely replicate the observed data and is also reasonably close to w0 . To this end, we consider a communication problem where a sender wishes to transmit a data set to a receiver using a message of the shortest possible length. However, unlike the standard MDL criterion, we assume that the initial model with w0 is known to the receiver. Namely, we try to send message length with respect to w0 − w, rather than with respect to w. Since we can avoid encoding parameter values equal to the initial ones, this metric prefers the initial model. The new parameters w0 − w are regarded as weights of the neural network, and their initial values are set to zero. Then, in order to obtain a learning result that is reasonably close to the initial model, the network is trained with weight decay, using a method called the MDL regularizer (Saito & Nakano, 1997).
Revising Models of Gene Regulation
63
When the modeling task includes some unobserved variables, like RR in Figure 1, we cannot directly revise links associated with those variables. To cope with such situations, our method adopts a simple forward-backward estimation based on the initial model. If x(i) is an unobserved variable, then its value can be estimated in the forward direction by the equation, x ˆ(i)(0) = j A(i, j)x(j)+ b(j). On the other hand, if S is a set of observed variables linked directly from x(i), i.e., S = {x(k) : k > i ∧ A(k, i) = 0}, then for x(k) ∈ S, the equation for the backward estimation is x(i) = A(k, i)−1 (x(k) − j=i A(k, j)x(j) − b(k)). This lets us estimate the values {ˆ x(i)(1) , ..., x ˆ(i)(M ) }, where M is the number of elements in S. Finally, our method estimates the value of x(i) as the average M of these values using the equation x ˆ(i) = (M + 1)−1 m=0 x ˆ(i)(m) . One could repeat these two procedures, estimation of the unobserved variables and revision of the parameters, although the current implementation makes only one pass. As stated above, our method performs gradient search through a space of parameters on causal links, with weight decay driving the model toward integer values. However, the resulting values are not strictly integers. To overcome this problem, in step 4 we employ a simple thresholding method. After sorting the resulting parameter values to predict one variable x(i), the system uses two thresholds, T−1 and T+1 , to divide this sorted list into three portions. Parameter value A(i, j) is set to −1 if A(i, j) < T−1 , to +1 if A(i, j) > T+1 , and to 0 otherwise. Note that T−1 ≤ T+1 , and we can obtain all possible integer lists with computational complexity O(N 2 ), where N denotes the number of parameters. Given these integer lists, our method selects the result that minimizes the MDL cost function defined by {0.5×s×log(M SE)}+{r×log(N )}, where s is the number of training samples, r is the number of revised parameters, and M SE is the mean squared error on the samples. The first term of the cost function is a code length for transmitting data, derived by assuming Gaussian noise for variables, while the second term is a code length for revision information, i.e., multiplication of the number of revised parameters and the cost of encoding an integer to indicate the parameter that is revised.
4
Experimental Studies of the Revision Method
In this section, we describe experimental studies of our revision method. We take a dual approach of evaluating the system using both natural data obtained from microarrays of Cyanobacteria cultures and synthetic data generated from known mathematical models. Natural data lets us evaluate the biological plausibility of changes suggested by our algorithm. However, because we have an extremely limited number of microarrays, it can be difficult to evaluate the reliability of the suggested revisions even if they appear biologically plausible. Therefore, we also used synthetic data to evaluate the robustness and reliability of our approach. Because we can generate synthetic data from a known model, we can measure the sensitivity and reliability of our algorithm in the presence of complicating factors such as errors in the initial model, small sample sizes, and noise.
64
K. Saito, S. Bay, and P. Langley
4.1
Revising the Model of Photosynthesis Regulation
We applied our method to revise the regulatory model of photosynthesis for wild type Cyanobacteria. We have microarray data which includes measurements for approximately 300 genes believed to play a role in photosynthesis. For this analysis, we focus on the genes in the model and do not consider links to other genes. The array data were collected at 0, 30, 60, 120, and 360 minutes after high light conditions were introduced, with four replicated measurements at each time point. We treat both RR and Photo, which represents the structure of the photosystem, as unmeasured variables. We currently treat the data as independent samples and ignore their temporal aspect, along with dependencies among the four replicates. We implemented our method in the C programming language and conducted all experiments on a 1.3 Ghz Pentium running Linux. Revising the photosynthesis model took 0.02 seconds of CPU time. For each variable, the observed values were normalized to a mean of zero and a standard deviation of one. Figure 2 shows the revised model, which reflects three changes: 1. dropping the link from dspA to RR; 2. connecting Photo to RR instead of psbA1 and psbA2; and 3. changing the sign of the link from PBS to Health from negative to positive. The first two changes are difficult to explain from a biological perspective. Because dspA is a light sensor, there should be either a direct or indirect path linking it with the genes cpcB, psbA1, or psbA2. Dropping the link disconnects dspA from those genes and removes it as possible cause. Also, the structure of the photosystem (Photo) is believed to depend on at least one of psbA1 or psbA2, and connecting Photo only to RR removes psbA1 and psbA2 as parents.2 Changing the sign of the link from PBS to Health is more plausible. The initial model was specified for high light conditions in which excessive light levels damage the organism. However, at lower light levels, increased PBS should aid the organism because it is a vital component in energy production. One explanation suggested by the microbiologist is that light levels during the biological experiment may not have been set correctly and were not high enough to reduce health. 4.2
Robustness of the Revision Approach
We evaluated the robustness of our approach by generating synthetic data from a known model and varying factors of interest. Specifically, we varied the number of training samples, the number of errors in the initial model, the observability of variables, and the noise level. We expected each of these factors to influence the behavior of the revision algorithm. 2
The genes psbA1 and psbA2 encode variants of the D1 protein, a necessary and central component of the Photosystem II reaction center (Wiklund et al., 2001).
Revising Models of Gene Regulation NBLR
+
NBLA
−
PBS
+ Light
+
dspA
− RR
− −
65
+
cpcB
Health
psbA2
Photo
−
+ psbA1
Fig. 2. Revised model of photosynthesis regulation in Cyanobacteria.
To this end, we generated training data by treating the structure of the model in Figure 1 as the true model. We assumed that each variable was a linear function of its parents with noise added from a random normal distribution. The root causal variable, Light, has no parents and was assigned a random uniform value between 0 and 1. We generated initial models to serve as starting points for revision by randomly adding links to, or deleting links from, the true model in Figure 1. Our dependent measure was the net number of corrections, that is, the number of correct changes minus the number of incorrect changes, suggested by the revision process. For each experimental condition, we generated 20 distinct training sets and averaged the results for this measure. Figure 3 (a) shows the results from one experimental condition that involved only observable variables and only a small amount of noise (σ = 0.1). The x axis in the graph represents the number of errors in the initial model, whereas the y axis specifies the net number of corrections. The three curves correspond to different size training sets, with the smallest containing only 25 instances and the largest involving 100 observations. In general, the revision method fared quite well, in that it consistently corrected almost all of the errors in the initial model. More data improved this performance, with 100 training cases being enough to give almost perfect results on all 20 runs. However, other factors can degrade the system’s behavior somewhat. Figure 3 (b) shows the results at the same noise level when the variable RR is unobservable but all others are available. Overall, the net number of corrections decreased substantially compared to the fully observable condition. However, the method still has enough power to recover portions of the true model. Figure 3 (c) and (d) show the system’s behavior with RR unobserved at higher levels of noise, with σ = 0.2 and σ = 0.4, respectively. The net number of corrections under these conditions is similar to that when σ = 0.1, which suggests that our approach is robust with respect to noise of this type. Note that σ = 0.4 constitutes a rather high noise level in comparison with the range of the variables (e.g., light varies from 0 to 1). We should also note that the system never suggested changes to the initial model when it was correct (i.e., contained zero errors). This indicates that the revision method is behaving in a conservative manner that is unlikely to make
66
K. Saito, S. Bay, and P. Langley
a good model worse, even in the presence of noise, unobservable variables, and small samples. This in turn suggests that our use of minimum description length is having the desired effect.
25 samples 50 samples 100 samples
25 samples 50 samples 100 samples
6
5
5
4
4
net corrections
net corrections
6
3 2 1
3 2 1 0
0 0
1
2 3 4 errors in initial model
5
6
0
1
(a)
6
5
6
25 samples 50 samples 100 samples
6
5
5
4
4
net corrections
net corrections
5
(b)
25 samples 50 samples 100 samples
6
2 3 4 errors in initial model
3 2 1
3 2 1
0
0 0
1
2 3 4 errors in initial model
5
6
0
(c)
1
2 3 4 errors in initial model
(d)
Fig. 3. Average net number of corrections to the initial model for 25, 50, and 100 samples when (a) all variables are observed and σ = 0.1, (b) the variable RR is unobserved and σ = 0.1, (c) RR is unobserved and σ = 0.2, and (d) RR is unobserved and σ = 0.4.
5
Directions for Future Research
The results from our experiments on Cyanobacteria data were disappointing, as they were difficult to explain from a biological perspective. However, on synthetic data our system was able to improve incorrect initial models even when there were few training samples, unobserved variables, and noise. This suggests that our general approach is feasible, but that we may need to address some of the limitations, chosen by design, in the approach. For instance,
Revising Models of Gene Regulation
67
we modeled the relationships between genes as a linear function. Although linear models are desirable because they have few parameters, they cannot model combinatorial effects among genes or thresholds in which a gene’s expression must be above a certain level before it can affect other genes. The neural network approach to revision is not limited to linear models and we could use a more general form to represent relationships between genes. We also restricted the genes that could appear in the model to a small subset of those measured by the microarray chips. The complete set of data contains about 300 variables from which we used the 11 variables present in the initial model. Restricting the number of variables involves a tradeoff. Including too many variables for the number of samples makes estimating relationships unreliable because of the multiple hypothesis testing problem (Shaffer, 1995). However, using too few variables increases the likelihood that we may have ignored an important variable from the analysis. Future implementations could minimize this problem by including an operator for adding new genes during the revision process and using domain knowledge to select only the most promising candidates for incorporation into the model. In addition, we should extend our approach to model revision in various other ways. Since transcriptional gene regulation takes time to occur, future systems should search through an expanded space of models that include time delays on links3 and feedback cycles. To handle more complex biological processes, it should also represent and revise models with subsystems that have little interaction with each other. Finally, each of these extensions would benefit from incorporation of additional biological knowledge, cast as taxonomies over both genes and regulatory processes, to constrain the search for improved models. Finally, we must test our approach on both more regulatory models and more microarray data before we can judge its practical value. Our biologist collaborators are collecting additional data on Cyanobacteria under more variable conditions, which we predict will provide additional power to our revision method. We also plan to evaluate the technique on additional data sets that we have acquired from other biologists, including ones that involve yeast development and lung cancer.
6
Related Research
Although most computational analyses of microarray data rely on clustering to group related genes, we are not the first to focus on inducing causal models of gene regulation. Most research on this topic encodes regulatory models as Bayesian networks with discrete variables (e.g., Friedman et al., 2000; Hartemink, 2002; Ong et al., 2002). Because microarray data are quantitative, this approach often includes a discretization step that may lose important information, whereas our approach deals directly with the observed continuous 3
An alternative is to model the regulation between genes with differential equations.
68
K. Saito, S. Bay, and P. Langley
values.4 These researchers also report methods that construct causal models from scratch, rather than revising an initial model, though some incorporate background knowledge to constrain the search process. An alternative approach represents hypotheses about gene regulation as linear causal models, which relate continuous variables through a set of linear equations. Such systems evaluate candidate models in terms of their ability to predict constraints among partial correlations, rather than their ability to predict the data directly. Within this framework, some methods (e.g., Saavedra et al., 2001) construct a linear causal model from the ground up, whereas others (e.g., Langley et al., 2002) instead revise an initial model, as in the approach we report here. One advantage of this constraint-based paradigm is that it can infer qualitative models directly, without the need to discretize or fit continuous parameters. In contrast, our technique combines search through a parameter space with weight decay to achieve a similar end. We should also mention approaches that, although not concerned with gene regulation, also construct causal models in scientific domains. One example comes from Koza et al. (2001), whose method formulates a quantitative model of metabolic processes from synthetic time series about chemical concentrations. Another involves Zupan et al.’s (2001) GenePath, which infers a qualitative genetic network to explain phenotypic results from gene knockout experiments. Mahidadia and Compton (2001) report an interactive system for revising qualitative models from experimental results in neuroendocrinology. Finally, our approach to revising scientific models borrows ideas from Saito et al. (2001), who transform an initial quantitative model into a neural network and utilize weight learning to improve its fit to observations.
7
Conclusions
In this paper, we characterized the task of discovering a qualitative causal model of gene regulation based on data from DNA microarrays. Rather than attempting to construct the model from scratch, we instead assume an existing model has been provided biologists who want to improve its fit to the data. These models require a causal ordering on variables, links between variables, and signs on these links. We presented an approach to this revision task that combines a hillclimbing search through the space of variable orderings and a gradient descent search for weights on links, with the latter using a weight decay method guided by minimum description length to drive weights to integer values. We illustrated the method’s behavior on a model of photosynthesis regulation in Cyanobacteria, using microarray data from biological experiments. However, our experimental evaluation also relied on synthetic data, which let us vary systematically the distance between the initial and target models, the amount of training data available, and the noise in these data. We found that the method scaled well on each of these dimensions, which suggests that it may prove a useful 4
Imoto et al. (2002) report one way to induce quantitative models of gene regulation within the framework of Bayesian networks.
Revising Models of Gene Regulation
69
tool for revising models based on biological data. We noted that our approach has both similarities to, and differences from, other recent techniques for inducing causal models of gene regulation. We must still evaluate the method on other data sets and extend it on various fronts, but our initial experiments on synthetic data have been encouraging.
Acknowledgements. This work was supported by the NASA Biomolecular Systems Research Program and by NTT Communication Science Laboratories, Nippon Telegraph and Telephone Corporation. We thank Arthur Grossman, Jeff Shrager, and C. J. Tu for the initial model, for microarray data, and for advice on biological plausibility.
References Friedman, N., Linial, M., Nachman, I., & Peer, D. (2000). Using Bayesian Networks to Analyze Expression Data. Journal of Computational Biology, 7 , 601–620. Grossman, A. R., Bhaya, D., & He, Q. (2001). Tracking the Light Environment by Cyanobacteria and the Dynamic Nature of Light Harvesting. The Journal of Biological Chemistry, 276 , 11449–11452. Hartemink, A. J., Gifford, D. K., Jaakkola, T. S., & Young, R. A. (2002). Combining Location and Expression Data for Principled Discovery of Genetic Regulatory Network Models. Pacific Symposium on Biocomputing, 7 , 437–449. Imoto, S., Goto, T., & Miyano, S. (2002). Estimation of Genetic Networks and Functional Structures Between Genes by using Bayesian Networks and Nonparametric Regression. Pacific Symposium on Biocomputing, 7 , 175–186. Koza, J. R., Mydlowec, W., Lanza, G., Yu, J., & Keane, M. A. (2001). Reverse engineering and automatic synthesis of metabolic pathways from observed data using genetic programming. Pacific Symposium on Biocomputing, 6 , 434–445. Langley, P., Shrager, J., & Saito, K. (in press). Computational discovery of communicable scientific knowledge. In L. Magnani, N. J. Nersessian, & C. Pizzi (Eds), Logical and computational aspects of model-based reasoning. Dordrecht: Kluwer Academic. Mahidadia, A., & Compton, P. (2001). Assisting model-discovery in neuroendocrinology. Proceedings of the Fourth International Conference on Discovery Science (pp. 214–227). Washington, D.C.: Springer. Ong, I. M., Glasner, J., & Page, D. (2002). Modeling Regulatory Pathways in E.Coli from Time Series Expression Profiles. Proceedings of the Tenth International Conference on Intelligent Systems for Molecular Biology. Rissanen, J. (1989). Stochastic complexity in statistical inquiry. World Scientific, Singapore. Saavedra, R., Spirtes, P., Scheines, R., Ramsey, J., & Glymour, C. (2001). Issues in Learning Gene Regulation from Microarray Databases. (Tech. Report No. IHMCTR-030101-01). Institute for Human and Machine Cognition, University of West Florida. Saito, K., Langley, P., Grenager, T., Potter, C., Torregrosa, A., & Klooster, S. A. (2001). Computational revision of quantitative scientific models. Proceedings of the Fourth International Conference on Discovery Science (pp. 336–349). Washington, D.C.: Springer.
70
K. Saito, S. Bay, and P. Langley
Saito, K., & Nakano, R. (1997). MDL regularizer: a new regularizer based on MDL principle. Proceedings of the 1997 International Conference on Neural Networks (pp. 1833–1838). Houston, Texas. Shaffer, J. P. (1995). Multiple Hypothesis Testing. Annual Review Psychology, 46, 561–584. Wiklund, R., Salih, G. F., Maenpaa, P., & Jansson, C. (2001) Engineering of the protein environment around the redox-active TyrZ in photosystem II. Journal of European Biochemistry , 268, 5356–5364. Zupan, B., Bratko, I., Demsar, J., Beck, J. R., Kuspa, A., Shaulsky, G. (2001). Abductive inference of genetic networks. Proceedings of the Eighth European Conference on Artificial Intelligence in Medicine. Cascais, Portugal.
SEuS: Structure Extraction Using Summaries Shayan Ghazizadeh and Sudarshan S. Chawathe University of Maryland Department of Computer Science College Park, MD {shayan,chaw}@cs.umd.edu
Abstract. We study the problem of finding frequent structures in semistructured data (represented as a directed labeled graph). Frequent structures are graphs that are isomorphic to a large number of subgraphs in the data graph. Frequent structures form building blocks for visual exploration and data mining of semistructured data. We overcome the inherent computational complexity of the problem by using a summary data structure to prune the search space and to provide interactive feedback. We present an experimental study of our methods operating on real datasets. The implementation of our methods is capable of operating on datasets that are two to three orders of magnitude larger than those described in prior work.
1
Introduction
In many data mining tasks, an important (and frequently most time-consuming) task is the discovery and enumeration of frequently occurring patterns, which are informally sets of related data items that occur frequently enough to be of potential interest for a detailed data analysis. The precise interpretation of this term depends on the data model, dataset, and application. Perhaps the best studied framework for data mining uses association rules to describe interesting relationships between sets of data items [AIS93]. In this framework, which is typically applied to market basket data (from checkout registers, indicating items purchased together), the critical operation is determining frequent itemsets, which are defined as sets of items that are purchased together often enough to pass a given threshold (called the support). For time series data, an analogous concept is a subsequence of the given series that occurs frequently. This paper defines an analogous concept, called frequent structures for semistructured data (represented as a labeled directed graph) and presents efficient methods for computing frequent structures in large datasets. Semistructured data is referred to data who has some structure, but is difficult to describe with a predefined, rigid schema. The structure of semistructured data is irregular, incomplete, frequently changing, and usually implicit or unknown to user. Common examples of this type of data include memos, Web pages, documentation, and bibliographies. Data mining is an iterative process in which a human expert refines the parameters of a data mining system based on intermediate results presented by the mining system. It is unreasonable to expect an expert to select the proper values for mining parameters a priori because such selection requires a detailed knowledge of the data, which is what the mining system is expected to enable. While frequent and meaningful feedback is S. Lange, K. Satoh, and C.H. Smith (Eds.): DS 2002, LNCS 2534, pp. 71–85, 2002. c Springer-Verlag Berlin Heidelberg 2002
72
S. Ghazizadeh and S.S. Chawathe
important for any data mining system, it is of particular importance when the data is semistructured because, in addition to the data-dependent relationships being unknown a priori, even the schema is not known (and not fixed). Therefore, rapid and frequent feedback to a human expert is a very important requirement for any system that is designed to mine semistructured data. Prior work (discussed in Section 4) on mining such data often falls short on this requirement. The main idea behind our method, which is called SEuS is the following threephase process: In the first phase (summarization), we preprocess the given dataset to produce a concise summary. This summary is an abstraction of the underlying graph data. Our summary is similar to data guides and other (approximate) typing mechanisms for semistructured data [GW97,BDFS97,NUWC97,NAM97]. In the second phase (candidate generation), our method interacts with a human expert to iteratively search for frequent structures and refine the support threshold parameter. Since the search uses only the summary, which typically fits in main memory, it can be performed very rapidly (interactive response times) without any additional disk accesses. Although the results in this phase are approximate (a supper set of final results), they are accurate enough to permit uninteresting structures to be filtered out. When the expert has filtered potential structures using the approximate results of the search phase, an accurate count of the number of occurrences of each potential structure is produced by the third phase (counting). Users are often willing to sacrifice quality for a faster response. For example, during the preliminary exploration of a dataset, one might prefer to get a quick and approximate insight into the data and base further exploration decisions on this insight. In order to address this need, we introduce an approximate version of our method, called L-SEuS. This method only returns the top-n frequent structures rather than all frequent structures. Due to space limitations we are not able to present the details of this approximate method here. Interested readers can refer to [GC02]. The methods in this paper have three significant advantages over prior work: First, they operate efficiently on datasets that are two to three orders of magnitude larger than those handled by prior work of which we are aware. Second, even for large datasets, our methods provide approximate results very quickly, enabling their use in an interactive exploratory data analysis. Third, for applications and scenarios that are interested in only the frequent structures, but not necessarily their exact frequencies, the most expensive counting phase can be completely skipped, resulting in great performance benefits. In order to evaluate our ideas, we have implemented our method in a data mining system for (semi)structured data (also called SEuS). In addition to serving as a testbed for our experimental study (Section 3), the system is useful in its own right as a tool for exploring (semi)structured data. We have found it to discover intuitively meaningful structures when applied to datasets from several domains. Our implementation of SEuS uses the Java 2 (J2SE) programming environment and is freely available at http://www.cs.umd.edu/projects/seus/ under the terms of the GNU GPL license. Figure 1 is a screenshot of our system in action. The current set of frequent structures is displayed together with a slider that allows the threshold to be modified. Given a new value for the threshold, the system computes (in interactive times) the new
SEuS: Structure Extraction Using Summaries
73
Fig. 1. A snapshot of SEuS system
set of frequent structures and presents them as depicted. We have found this iterative process to be very effective in arriving at interesting values of the parameter. The rest of this paper is organized as follows: In Section 2, we define the structure discovery problem formally and present our three-phase solution called SEuS. Sections 2.1, 2.2, and 2.3 describe the summarization, candidate generation, and counting phases. Section 3 summarizes the results of our detailed experimental study. Related work is discussed in Section 4 and we conclude in Section 5.
2
Structure Discovery
SEuS represents semistructured data as a labeled directed graph. In this representation, objects are mapped to vertices and relations between these objects are modeled by edges. A structure is defined to be a connected graph that is isomorphic to at least one subgraph of the database. Figure 2 illustrates the graph representation of a small XML database. Any subgraph of the input database that is isomorphic to a structure is called an instance of that structure. The number of instances of a structure is called the structure’s support. (We allow the instances to overlap.) For the data graph in Figure 2, a structure and its three instances are shown in Figure 3. We say a structure is T-frequent if it has a support higher than a given threshold T . Problem statement (frequent structure discovery): Given the graph representation of a database and a threshold T , find the set of T-frequent structures.
74
S. Ghazizadeh and S.S. Chawathe author
a1 child book child
child
child child
n2
p3 paper
name
b1
child
child
child cite
child
t6 title
y1
author a2 child year idref
t1 title book child
b4 child
name child
v2
y4
t4
volume
year
title
p1 paper
child
cite
v1 y3
j1
year journal cite child
b2 book child
child
child child
idref
n1 cite
book
b3
volume
p2
t3 title
child
child
t5
c1
title
conference
t7 title
child
t2
y2
title
year
Fig. 2. Example input graph
A naive approach for finding frequent structures consists of enumerating all subgraphs, partitioning this set of subgraphs into classes based on graph isomorphism, and returning a representative from the classes with cardinality greater than the support threshold. Unfortunately, the number of subgraphs of a graph database is exponential in the size of the graph. Further, the naive approach tests each pair of these subgraphs for isomorphism in worst case. Although graph isomorphism is not known to be NP-hard (or in P) [For96], it is a difficult problem and an approach relying on an exponential number of isomorphism tests is unlikely to be practical for large databases. Given the above, practical systems must use some way to avoid examining all the possible subgraphs and must calculate the support of structures without partitioning the set of all possible subgraphs. Instead of enumerating all of the subgraphs in the beginning, we can use a level-by-level expansion of subgraphs similar to the k-itemset approach adopted in Apriori [AS94] for market basket data. We start from subgraphs of size one (single vertex) and try to expand them by adding more vertices and edges. A subgraph is not expanded anymore as soon as we can reason that its support will fall under the threshold based on downward closure property: A structure has a support higher than a threshold if all of its subgraphs also have a support higher than the threshold. A number of systems have used such a strategy for discovering frequent structures [IWM00,KK01,CH00] along other heuristics to speed up the process. (See Section 4 for details.) However, The results reported in these papers, as well as our experiments, suggest that these methods do not scale to very large databases. The main factor hurting performance of these methods is the need to go through the database to determine the support of each structure. Although the number of structures for which the support has to be calculated has decreased significantly compared to the
SEuS: Structure Extraction Using Summaries
book child
child
child
-
year
t1
title Structure
child t2
b2
child y1
Subgraph 1
child
child
y2
t4
Subgraph 2
b1
75
b4
child y4
Subgraph 3
Fig. 3. A structure and its three instances
naive approach (due to the use of downward closure properties and other heuristics), the calculation of the support of the remaining structures is still expensive. Further, all of these systems operate in a batch mode: After providing the input database, a user has to wait for the structure discovery process to terminate before any output is produced. There are no intermediate (partial or approximate) results, making exploratory data analysis difficult. This batch mode operation can cause major problems, especially when the user does not have enough domain knowledge to guess proper values for mining parameters (e.g., support threshold). In order to operate efficiently, SEuS uses data summaries instead of the database itself. Summaries provide a concise representation of a database at the expense of some accuracy. This representation allows our system to approximate the support of a structure without scanning the database. We also use the level-by-level expansion method to discover frequent structures. SEuS has three major phases: The first phase (summarization) is responsible for creating the data summary and is described in Section 2.1. In the second phase (candidate generation), SEuS finds all structures that have an estimated support above the given threshold; it is described in Section 2.2. The second phase reports such candidate structures to the user, and this early feedback is useful for exploratory work. The exact support of structures is determined in the third phase (counting), described in Section 2.3. 2.1
Summarization
We use a data summary to estimate the support of a structure (i.e., the number of subgraphs in the database that are isomorphic to the structure). Our summary is similar in spirit to representative objects, graph schemas, and DataGuides [NUWC97,BDFS97, GW97]. The summary is a graph with the following characteristics. For each distinct vertex label l in the original graph G, the summary graph S has an l-labeled vertex. For each m-labeled edge (v1 , v2 ) in the original graph there is an m-labeled edge (l1 , l2 )
76
S. Ghazizadeh and S.S. Chawathe
in S, where l1 and l2 are the labels of v1 and v2 , respectively. The summary S also associates a counter with each vertex (and edge) indicating the number of vertices (respectively, edges) in the original graph that it represents. For example, Figure 4 depicts the summary generated for the input graph of Figure 2.
name
idref:2 child:2 child:2
2 child:1 author
2
book 4
cite:2 child:2
cite:1 child:1 paper child:1
cite:1child:4 child:4 2 child:3 volume 7
3
title
child:1 child:1
1
1 journal conference
4 year
Fig. 4. Summary graph
Since all vertices in the database with the same label map to one vertex in the summary, the summary is typically much smaller than the original graph. For example, the graph of Figure 2 has four vertices labeled book, while the summary has only one vertex representing these four vertices. In this simple example, the summary is only slightly smaller than the original data. However, as noted in [GW97], many common datasets are characterized by a high degree of structural replication, giving much greater space savings. (For details, see Table 1 in Section 3.) These space savings come at the cost of reduced accuracy of representation. In particular this summary tells us the labels on possible edges to and from the vertices labeled paper, although they may not all be incident on the same vertex in the original graph. (For example, journal and conference vertices never connect to the same paper vertex, but the summary does not contain this information.)
child, title 1 child, con f erence
child, journal child, year
child, title
cite, book
1
child, title 1
Fig. 5. Counting Lattice for paper vertex
SEuS: Structure Extraction Using Summaries
77
We can partly overcome this problem by creating a richer summary. Instead of storing only the set of edges leaving a vertex label and their frequencies, we can create a counting lattice (similar to the one used in [NAM97]), L(v) for each vertex v. For every distinct set of edges leaving v, we create a node in L(v) and store the frequency of this set of outgoing edges. For example, consider the vertex label paper in Figure 2. The counting lattice for this vertex is depicted in Figure 5. In the input graph, there are three different types of paper vertices with respect to their outgoing edges. One of them, p3 , has a single outgoing edge labeled child leading to a title vertex. Another instance, p2 , has two outgoing edges to title and conference vertices. Finally, p1 has four outgoing edges. The lattice represents these three types of vertices with label paper separately, while a simple summary does not distinguish between them. Each node in lattice also stores the support of the paper vertex type it represents. We call the original summary a level-0 summary and the summary obtained by adding this lattice structure a level-1 summary. Using the level-1 summary, we can reason that there is no paper vertex in the database that connects to both journal and conference vertices, which is not possible using only level-0 summary. This process of enriching the summary by differentiating vertices based on the labels of their outgoing edges can be carried further by using the labels of vertices and edges that are reachable using paths of lengths two or more. We refer to such summaries as level-k summaries: A level-k summary differentiates vertices based on labels of edges and vertices on outgoing paths of length k. However, building level-k summaries for k ≥ 2 is considerably more difficult than building level-0 and level-1 summaries. Level-0 summaries are essentially data guides, and level-1 summaries can be built with no additional cost if the file containing the graph edges is sorted by the identifiers of source vertices. For summaries of higher levels, additional passes of graph are required. Further, our experiments show that level-1 summaries are accurate enough for the datasets we study (See [GC02] for details), so the additional benefit of higher summary levels is unclear. In the rest of this paper, we focus on level-0 and level-1 summaries. We assume that the graph database is stored on disk as a sequence of edges, sorted in lexicographic order of the source vertex. Clearly, building level-0 summary requires only a single sequential scan of the edges file. We build the summary incrementally in memory as we scan the file. For an edge (v1 , v2 , l) we increment the counters associated with the summary nodes representing the labels l1 and l2 of v1 and v2 , respectively. Similarly, the counter associated with the summary edge (s(l1 ), s(l2 ), l) is incremented, where s(li ) denotes the summary node representing label li . (If the summary nodes or edges do not exist, they are created.) Since the edges file is sorted in lexicographic order of the source, we can be sure that we get all of the outgoing edges of a vertex before encountering another source vertex. Therefore, we can create the level-1 summary in the same pass as we build the level-0 summary. We use a level-0 summary L0 to estimate the support of a structure S as follows: By construction, there is at most one subgraph of L0 (say, S ) that is isomorphic to S. If no such subgraph exists, then the estimated (and actual) support of S is 0. Otherwise, let C be the set of counters on S (i.e., C consists of counters on the nodes and edges of S ). The support of S is estimated by the minimum value in C. Given our construction of the summary, this estimate is an upper bound on the true support of S. With a level-1
78
S. Ghazizadeh and S.S. Chawathe
summary L1 , we estimate the support of a structure S as follows: For each vertex v of S, let L(v) be the set of lattice nodes in L1 that represent a set of edges that is a superset of the set of out-edges of v. Let c(v) denote the sum of the counters for nodes in L(v). The support of S is estimated to be minv∈S c(v). This estimate is also an upper bound on the true support of S. Further, it is a tighter bound than that given by the corresponding level-0 summary. 2.2
Candidate Generation
A simplified version of our candidate generation algorithm is outlined in Figure 6: CandidateGeneration(x) returns a list of candidate structures whose estimated support is x or higher. It maintains two lists of structures: open and candidate. In the open list we store structures that have not been processed yet (and that will be checked later). The algorithm begins by adding all structures that consist of only one vertex and pass the support threshold test to the open list. The rest of the algorithm is a loop that repeats until there are no more structures to consider (i.e., the open list is empty.) In each iteration, we select a structure (S) from the open list and we use it to generate larger structures (called S’s children) by calling the expand subroutine, described below. New child structures that have an estimated support of at least x are added to the open list. The qualifying structures are accumulated in the candidate result, which is returned as the output when the algorithm terminates. Algorithm CandidateGeneration(threshold) 1. candidate ←∅; open ←∅; 2. for v ∈ summary and support(v) ≥ threshold 3. do create a structure s consisting of a single vertex v; 4. open ←open ∪s; 5. while open = ∅ 6. do S ←any structure in open; 7. open ←open −S; candidate ←candidate ∪S; 8. children ←expand(S); 9. for c ∈ children 10. do if support(c) ≥ threshold and c ∈ candidate 11. then open ←open ∪c; Fig. 6. Simplified Candidate Generation Algorithm
Given a structure S, the expand subroutine produces the set of structures generated by adding a single edge to S (termed the children of S). In the following description of the expand(S) subroutine, we use S(v) to denote the set of vertices in S that have the same label as vertex v in the data graph and V (s) to denote the set of data vertices that have the same label as a vertex s in S. For each vertex s in S, we create the set addable(S, s) of edges leaving some vertex in V (s). This set is easily determined from the data summary: It is the set of out-edges for the summary vertex representing s. (As we shall discuss in Section 3, this ability to generate structures using only the in-memory
SEuS: Structure Extraction Using Summaries
79
summary instead of the disk resident database results in large savings in running time.) Each edge e = (s, v, l) in addable(S, s) that is not already in S is a candidate for expanding S. If S(v) (the set of vertices with the same label as e’s destination vertex) is empty, we add a new vertex x with the same label as v and a new edge (s, x, l) to S. Otherwise, for each x ∈ S(v) if (s, x, l) in not in S, a new structure is created from S and e by adding the edge (s, x, l) (an edge between vertices already in S). If s does not have an l-labeled edge to any of the vertices in S(v), we also add a new structure which is obtained from S by adding a vertex x with the same label as v and an edge (s, x , l). For example consider the graph in Figure 2. Let us assume that we want to expand a structure S consisting of a single vertex s labeled author. The set addable(S, s) is child idref child child {author −→ book, author −→ book, author −→ name, author −→ paper} (all the edges that leave an author labeled vertex in database). Since S has only one vertex, it can be expanded only by adding these four edges. Using the first edge in the addable set, a new structure is obtained from S by adding a new book-labeled vertex and connecting s to this new vertex by a child edge. The other edges in addable(S, s) give rise to three other structures in this manner. 2.3
Support Counting
Once the user is satisfied with the structures discovered in the candidate generation phase, she may be interested in finalizing the frequent structure list and getting the exact support of the structures. (Recall that the candidate generation phase provides only a quick, approximate support for each structure, based on the in-memory summary.) This task is performed in the support counting phase, which we describe here. Let us define the size of a structure to be the number of nodes and edges it contains; we refer to a structure of size k as a k-structure. From the method used for generating candidates (Section 2.2), it follows that for every k-structure S in the candidate list there exists a structure Sp of size k − 1 or k − 2 in the candidate list such that Sp is a subgraph of S. We refer to Sp as the parent of S in this context. Clearly, every instance I of S has a subgraph I that is an instance of Sp . Further, I differs from I only in having one fewer edge and, optionally, one fewer vertex. We use these properties in the support counting process. Determining the support of a 1-structure (single vertex) consists of simply counting the number of instances of a like-labeled vertex in the database. During the counting phase, we store not only the support of each structure (as it is determined), but also a set of pointers to that structure’s instances on disk. To determine the support of a k-structure S for k > 1, we revisit the instances of its parent Sp using the saved pointers. For each such instance I , we check whether there is a neighboring edge and, optionally, a node that, when added to I generates an instance I of S. If so, I is recorded as an instance of S. This operation of growing an instance I of Sp to an instance I of S is similar to the expand operation used in the candidate generation phase; however, there are two difference. First, in the counting phase we expand subgraphs of the database whereas in the candidate generation phase we expand abstract structures without referring to the disk-resident data (using only the summary). Second, in the counting phase we need to find an edge or vertex in the database to be added to the instance that satisfies the
80
S. Ghazizadeh and S.S. Chawathe
constraints imposed by the expansion which created the structure (e.g., the label of the edge). Whereas in the candidate generation phase, we add any possible edges and vertices to the structure.
3
Experimental Evaluation
In order to evaluate the performance of our method we have performed a number of experiments. We have implemented SEuS using the Java 2 (J2SE) programming environment. For graph isomorphism tests, we have used the nauty package[McK02] to derive canonically labeled isomorphic graphs. Since we have two levels of summaries, we append a “-Sd” to a system’s name to show which level of summary has been used with the method in a particular experiment (e.g., SEuS-S0 is the SEuS method using summary level-0). In the experiments described below, we have used a PC-class machine with a 900 MHz Intel Pentium III processor and one gigabyte of RAM, running the RedHat 7.1 distribution of GNU/Linux. Where possible, we have compared our results with those for SUBDUE version 4.3 (serial version), which is implemented in the C programming language. Due to space restictions we are not able to present detailed experimental results here. Extensive results have been presented in [GC02]. Table 1 presents some characteristics of the 13 datasets we have used for our experiments. Table 1. Datasets used in experiments Name Credit-* Diabetes-* Vote Chemical Chess Medical-*
Description
Vertex Edge Summary Graph labels labels Size Size Credit card applications 59 20 136 3899-27800 Diabetes patient records 7 8 39 4556-8500 Congressional voting records 4 16 52 8811 Chemical compounds 66 4 338 18506 Chess relational domain 7 12 88 189311 Medical publication citations 75 4 175 4M-10M
Figure 7 compares the running time of SEuS and SUBDUE on the 13 datasets of Table 1. Running times of SEuS using both levels of summaries are depicted here. It is important to notice that SEuS versions run for a longer time compared to SUBDUE because it is looking for all frequent structures, whereas SUBDUE only returns the n most frequent structures (n = 5 in these experiments.). The running times of SEuS increases monotonically as the size of datasets increases. The irregularities in the running time of SUBDUE are due to the fact that, besides the size of a dataset, factors such as the number of vertex and edge labels have a significant effect on the performance of SUBDUE. Referring to Table 1, it is clear that Credit datasets have many more labels than the Diabetes datasets. Although Credit-1 and Credit-2 datasets are smaller than the Diabetes datasets, it takes SUBDUE longer to mine them because it tries to expand the subgraphs by all possible edges at each iteration. Then SUBDUE decides which isomorphism class is better by considering the number of subgraphs in them and the
SEuS: Structure Extraction Using Summaries
81
size of the subgraphs. (In SUBDUE the sets of isomorphic subgraphs are manipulated as bags of subgraphs.) When there is a large number of different vertex or edge labels, there will be a larger number of subgraphs to choose between and since SUBDUE accesses the database for each subgraph, the running time increases considerably. The number of edge or vertex labels affects SEuS in a similar way, but since we do not access the main database to find the support of a structure (we use the summary instead) this number does not significantly affect our running time. SEuS has a phase of data summary generation which SUBDUE does not perform. In small datasets this additional effort is comparable to the overall running time. Also note that the running time of SEuS increases if we use level-1 summary instead of level-0 summary. This increase in running time is mainly due to the overhead of creating a richer summary. This additional effort will result in more accurate results (lower overestimation which yeilds less waste of time in the counting phase). We are comparing a Java implementation of SEuS with the C implementation of SUBDUE. While the difference in efficiency of these programming environments is not significant for large datasets, it is a factor for the smaller ones.
Fig. 7. Running time
As the datasets grow, the running time of SUBDUE grows very quickly, while SEuS does not show such a sharp increase. With our experimental setup, we were unable to obtain any results from SUBDUE for datasets larger than 3 MB (after running for 24 hours). For this reason, Figure 7 presents the running time of only SEuS method for the large datasets. To best of our knowledge, other complete structure discovery methods cannot handle datasets with sizes comparable to those we have used here. AGM and FSG methods, presented in [IWM00,KK01], take respectively eight days and 600 seconds to process the Chemical dataset, for which SEuS only needs 20 seconds[KK01].
82
S. Ghazizadeh and S.S. Chawathe
(Unfortunately, we were unable to obtain the FSG system to perform a more detailed comparison.) One should note that for very small thresholds, these methods will have a better performance because with those thresholds a large number of structures will be frequent and our summary does not provide a significant pruning while introducing the overhead of creating a summary. As discussed in Section 1, the SEuS system provides real-time feedback to the user by quickly displaying the frequent structures resulting from different choices of the threshold parameter. This interactive feedback is possible because the time spent in the candidate generation (search) phase is very small. Figure 8 justifies this claim. It depicts the percentage of time used by each of the three phases in processing different datasets. As datasets get larger, the fraction of running time spent on summarizing the graph falls rapidly. Also the time spent in the candidate generation phase is relatively small. Therefore, our strategy of creating the summary once and running the candidate generation phase multiple times with different input parameters (in order to determine suitable values before proceeding to the expensive counting phase) is very effective.
Percentage of Time (%)
1 Count Search Summarize
2 Cr
n 1 2 3 4 -1 -2 -1 -2 -3 -4 it it es es te it it ss io l- l- l- led red abet abet Vo red red Che viat dica dica dica dica C Di Di C C A Me Me Me Me
Datasets (Increasing size) Fig. 8. Time spent in algorithm phases
4
Related Work
Much of the prior work on structure discovery is domain dependent (e.g., [Win75,Lev84, Fis87,Leb87,GLF89,CG92]) and a detailed comparison of these methods appears in [Con94]. We consider only domain independent methods in this paper. The first such system, CLIP, discovers patterns in graphs by expanding and combining patterns discovered in previous iterations [YMI93]. To guide the search, CLIP uses an estimate of the compression resulting from an efficient representation of repetitions of a candidate structure. The estimate is based on a linear-time approximation for graph isomorphism. SUBDUE [CH00] also performs structure discovery on graphs. It uses the minimum description length principle to guide its beam search. SUBDUE uses an inexact graph matching algorithm during the process to find similar structures. SUBDUE discovers structures differently from CLIP. First, SUBDUE produces only single structures evaluated using minimum description length, whereas CLIP produces a set of structures that collectively compress the input graph. CLIP has the ability to grow structures using the merge operator between two previously found structures,
SEuS: Structure Extraction Using Summaries
83
while SUBDUE only expands structures one edge at a time. Our system is similar to SUBDUE with respect to structure expansion. Second, CLIP estimates the compression resulting from using a structure, but SUBDUE performs an expensive exact measurement of compression for each new structure. This expensive task causes the SUBDUE system to be very slow when operating on large databases. AGM [IWM00] is an Apriori-based algorithm for mining frequent structures which are induced subgraphs of the input graph. The main idea is similar to that used by the market basket analysis algorithm in [AS94]: a (k + 1)-itemset is a candidate frequent itemset only if all of its k-item subsets are frequent. In AGM, a graph of size k + 1 is considered to be a candidate frequent structure only if all its subgraphs of size k are frequent. AGM only considers the induced subgraphs to be candidate frequent structures. (Given a graph G, subgraph Gs is called an induced subgraph if V (Gs ) ⊂ V (G), E(Gs ) ⊂ E(G) and ∀u, v ∈ V (Gs ), (u, v) ∈ E(Gs ) ⇔ (u, v) ∈ E(G).) This restriction reduces the size of the search space, but also means that interesting structures that are not induced subgraphs cannot be detected by AGM. After producing the next generation of candidate frequent structures, AGM counts the frequency of each candidate by scanning the database. As in SUBDUE, this need for a database scan at each generation limits the scalablity of this method. FSG [KK01] is another system that finds all connected subgraphs that appear frequently in a large graph database. Similar to AGM, this system uses the level-by-level expansion adopted in Apriori. The key features of FSG compared to AGM are the following: (1) it uses a sparse graph representation which minimizes storage and computation, (2) there is no restriction on the structure’s topology (e.g., induce subgraph restriction) other than their connectivity, and (3) it incorporates a number of optimizations for candidate generation and counting which makes it more scalable (e.g., transaction ID lists for counting). However, this system still scans the database in order to find the support of next generation structures. The experimental results in [KK01] show that FSG is considerably faster than AGM. One should note that AGM and FSG both operate on a transaction database where each transaction is a graph, so that their definition of a frequent structure’s support can be applicable. In SEuS we do not have this restriction, and SEuS can be applied to both a transaction database and a large connected graph database. As mentioned in Section 3, for a common Chemical dataset, FSG needs 600 seconds, where SEuS returned the frequent structures in less than 20 seconds. Asai [AAK+ 02] proposes FREQT algorithm for discovering frequent structures in semistructured data. FREQT models semistructured data and the frequent structures using labeled ordered trees. The key contribution of this work is the notion of the rightmost expansion, a technique to grow a tree by attaching new nodes only to the rightmost branch of the tree. The authors show that it is sufficient to maintain only the instances of the rightmost leaf to efficiently implement incremental computation of structure frequency. Limiting the search space to ordered trees allows the method to scale almost linearly in the total size of maximal tree contained in the collection. In [CYLW02], authors propose another method for frequent structure discovery in semistructured collections. In this work, the dataset is a collection of semistructured objects treated as transactions similar to FSG method. Motivated by the semistructured data path expressions, the authors try to represent the objects and patterns as a set
84
S. Ghazizadeh and S.S. Chawathe
of labeled paths which can include wildcards. After introducing the notion of weaker than for comparing a structure path set with a transaction object, the algorithm tries to discover the set of all patterns that have a frequency higher than a given threshold. The authors discuss that the methods is motivated and well-suited for collections consisting of similarly structured objects with minor differences. The problem of finding frequent structures is related to the problem of finding implicit structure (or approximate typing) in semistructured databases [NAM97,NAM98]. In type inference, the structures are typically limited to rooted trees and each structure must have a depth of one. Further, the frequency of a structure is not the only metric used in type inference. For instance, a type that occurs infrequently may be important if its occurrences have a very regular structure. Despite these differences, it may be interesting to investigate the possibility of adapting methods from one problem for the other.
5
Conclusion
In this paper, we motivated the need for data mining methods for large semistructured datasets (modeled as labeled graphs with several million nodes and edges). We focused on an important building block for such data mining methods: the task of finding frequent structures, i.e., structures that are isomorphic to a large number of subgraphs of the input graph. We presented the SEuS method, which finds frequent structures efficiently by using a structural summary to estimate structure support. Our method has two main distinguishing features: First, due to their use of a summary data structure, they can operate on datasets that are two to three orders of magnitude larger than those used by prior work. Second, our methods provide rapid feedback (delay of a few seconds) in the form of candidate structures, thus permitting their use in an interactive data exploration system. As ongoing work, we are exploring the application of our methods to finding association rules and other correlations in semistructured data. We are also applying our methods to the problems of classification and clustering by using frequent structures to build a predictive model.
References [AAK+ 02]
[AIS93] [AS94]
[BDFS97]
Tatsuya Asai, Kenji Abe, Shinji Kawasoe, et al. Efficient substructure discovery from large semi-structured data. In Proc. of the Second SIAM International Conference on Data Mining, 2002. R. Agrawal, T. Imielinski, and A. Swami. Mining associations between sets of items in massive databases. SIGMOD Record, 22(2):207–216, June 1993. R. Agrawal and R. Srikant. Fast algorithms for mining association rules. In Proc. of the 20th International Conference Very Large Data Bases, pages 487–499. Morgan Kaufmann, 1994. P. Buneman, S. B. Davidson, M. F. Fernandez, and D. Suciu. Adding structure to unstructured data. In Proc. of the 6th International Conference on Database Theory, 1997.
SEuS: Structure Extraction Using Summaries [CG92] [CH00] [Con94] [CYLW02]
[Fis87] [For96] [GC02]
[GLF89] [GW97]
[IWM00]
[KK01] [Leb87] [Lev84] [McK02] [NAM97] [NAM98]
[NUWC97]
[Win75] [YMI93]
85
D. Conklin and J. Glasgow. Spatial analogy and subsumption. In Proc. of the Ninth International Conference on Machine Learning, pages 111–116, 1992. D. J. Cook and L. B. Holder. Graph-based data mining. ISTA: Intelligent Systems & their applications, 15, 2000. D. Conklin. Structured concept discovery: Theory and methods. Technical Report 94-366, Queen’s University, 1994. Gao Cong, Lan Yi, Bing Liu, and Ke Wang. Discovering frequent substructures from hierarchical semi-structured data. In Proc. of the Second SIAM International Conference on Data Mining, 2002. D. H. Fisher, Jr. Knowledge acquisition via incremental conceptual clustering. Machine Learning, (2):139–172, 1987. S. Fortin. The graph isomorphism problem. Technical Report 96-20, University of Alberta, 1996. Shayan Ghazizadeh and Sudarshan Chawathe. Discovering freuqent structures using summaries. Technical report, University of Maryland, Computer Science Department, 2002. J. H. Gennari, P. Langley, and D. Fisher. Models of incremental concept formation. Artificial Intelligence, (40):11–61, 1989. R. Goldman and J. Widom. Dataguides: Enabling query formulation and optimization in semistructured databases. In Proc. of the Twenty-Third International Conference on Very Large Data Bases, pages 436–445, 1997. A. Inokuchi, T. Washio, and H. Motoda. An apriori-based algorithm for mining frequent substructures from graph data. In Proc. of the 4th European Conference on Principles and Practice of Knowledge Discovery in Databases, pages 13–23, 2000. M. Kuramochi and G. Karypis. Frequent subgraph discovery. In Proc. of the 1st IEEE Conference on Data Mining, 2001. M. Lebowitz. Experiments with incremental concept formation: Unimem. Machine Learning, (2):103–138, 1987. R. Levinson. A self-organizing retrieval system for graphs. In Proc. of the National Conference on Artificial Intelligence, pages 203–206, 1984. B. D. McKay. nauty user’s guide (version 1.5), 2002. S. Nestorov, S. Abiteboul, and R. Motwani. Inferring structure in semistructured data. In Proc. of the Workshop on Management of Semistructured Data, 1997. S. Nestorov, S. Abiteboul, and R. Motwani. Extracting schema from semistructured data. In Proc. of the ACM SIGMOD International Conference on Management of Data, pages 295–306, 1998. S. Nestorov, J. Ullman, J. Wiener, and S. Chawathe. Representative objects: Concise representations of semistructured, hierarchial data. In Proc. of the International Conference on Data Engineering, pages 79–90, 1997. P. H. Winston. Learning structural descriptions from examples. In The Psychology of Computer Vision, pages 157–209. 1975. K. Yoshida, H. Motoda, and N. Indurkhya. Unifying learning methods by colored digraphs. In Proc. of the International Workshop on Algorithmic Learning Theory, volume 744, pages 342–355, 1993.
Discovering Best Variable-Length-Don’t-Care Patterns Shunsuke Inenaga1 , Hideo Bannai3 , Ayumi Shinohara1,2 , Masayuki Takeda1,2 , and Setsuo Arikawa1 1
Department of Informatics, Kyushu University 33, Fukuoka 812-8581, Japan 2 PRESTO, Japan Science and Technology Corporation (JST) {s-ine, ayumi, takeda, arikawa}@i.kyushu-u.ac.jp 3 Human Genome Center, University of Tokyo, Tokyo 108-8639, Japan [email protected]
Abstract. A variable-length-don’t-care pattern (VLDC pattern) is an element of set Π = (Σ ∪ {})∗ , where Σ is an alphabet and is a wildcard matching any string in Σ ∗ . Given two sets of strings, we consider the problem of finding the VLDC pattern that is the most common to one, and the least common to the other. We present a practical algorithm to find such best VLDC patterns exactly, powerfully sped up by pruning heuristics. We introduce two versions of our algorithm: one employs a pattern matching machine (PMM) whereas the other does an index structure called the Wildcard Directed Acyclic Word Graph (WDAWG). In addition, we consider a more generalized problem of finding the best pair q, k, where k is the window size that specifies the length of an occurrence of the VLDC pattern q matching a string w. We present three algorithms solving this problem with pruning heuristics, using the dynamic programming (DP), PMMs and WDAWGs, respectively. Although the two problems are NP-hard, we experimentally show that our algorithms run remarkably fast.
1
Introduction
A vast amount of data is available today, and discovering useful rules from those data is quite important. Very commonly, information is stored and manipulated as strings. In the context of strings, rules are patterns. Given two sets of strings, often referred to as positive examples and negative examples, it is desired to find the pattern that is the most common to the former and the least common to the latter. This is a critical task in Discovery Science as well as in Machine Learning. A string y is said to be a substring of a string w if there exist strings x, z ∈ Σ ∗ such that w = xyz. Substring patterns are possibly the most basic patterns to be used for the separation of two sets S, T of strings. Hirao et al. [8] stated that such best substrings can be found in linear time by constructing the suffix tree for S ∪ T [12,21,7]. They also considered subsequence patterns as rules for separation. A subsequence pattern p is said to match a string w if p can be obtained by removing zero or more characters from w [2]. Against the fact that finding S. Lange, K. Satoh, and C.H. Smith (Eds.): DS 2002, LNCS 2534, pp. 86–97, 2002. c Springer-Verlag Berlin Heidelberg 2002
Discovering Best Variable-Length-Don’t-Care Patterns
87
the best subsequence patterns to separate given two sets of strings is NP-hard, they proposed an algorithm to solve the problem with practically reasonable performance. More recently, an efficient algorithm to discover the best episode patterns was proposed in [9]. An episode pattern p, k, where p is a string and k is an integer, is said to match a string w if p is a subsequence of a substring u of w with |u| ≤ k [14,6,20]. The problem to find the best episode patterns is also known to be NP-hard. In this paper, we focus on a pattern containing a wildcard that matches any string. The wildcard is called a variable length don’t care and is denoted by . A variable-length-don’t-care pattern (VLDC pattern) is an element of Π = (Σ ∪ { })∗ , and is also sometimes called a regular pattern as in [19]. When a, b ∈ Σ, ab bb ba is an example of a VLDC pattern and, for instance, matches string abbbbaaaba with the first and second ’s replaced by b and aaa, respectively. The language L(q) of a pattern q ∈ Π is the set of strings obtained by replacing ’s in q with arbitrary strings. Namely, L(ab bb ba) = {abubbvba | u, v ∈ Σ ∗ }. The class of this language corresponds to a class of the pattern languages proposed by Angluin [1]. VLDC patterns are generalization of substring patterns and subsequence patterns. For instance, consider a pattern string abc ∈ Σ ∗ . The substring matching problem corresponding to the pattern is given by the VLDC pattern abc . Also, the VLDC pattern a b c leads to the subsequence pattern matching problem. This paper is devoted to introducing a practical algorithm to discover the best VLDC pattern to distinguish two given sets S, T of strings. To speed up the algorithm, firstly we restrict the search space by means of pruning heuristics inspired by Morishita and Sese [16]. Secondly, we accelerate the matching phase of the algorithm in two ways, as follows: In [11], we introduced an index structure called the Wildcard Directed Acyclic Word Graph (WDAWG). The WDAWG for a text string w recognizes all possible VLDC patterns matching w, and thus enables us to examine whether a given VLDC pattern q matches w in O(|q|) time. More recently, a space-economical version of its construction algorithm was presented in [10]. We use WDAWGs for quick matching of VLDC patterns. Another approach is to preprocess a given VLDC pattern q, building a DFA accepting L(q). We use it as a pattern matching machine (PMM) which runs over a text string w and determines whether or not q matches w in O(|w|) time. We furthermore propose a generalization of the VLDC pattern matching problem. That is, we introduce an integer k called the window size which specifies the length of an occurrence of a VLDC pattern that matches w ∈ Σ ∗ . The introduction of k leads to the generalization of the episode patterns as well. Specifying the length of an occurrence of a VLDC pattern is of great significance especially when classifying long strings over a small alphabet, since a short VLDC pattern surely matches most long strings. Therefore, for example, when two sets of biological sequences are given to be separated, this approach is adequate and promising. Pruning heuristic to speed up our algorithm finding the best pair q, k is also presented. We propose three approaches effective in computing the best pair, using the dynamic programming, PMMs, and WDAWGs, respectively.
88
S. Inenaga et al.
We declare that this work generalizes and outperforms the ones accomplished in [8,9], since it is capable of discovering more advanced and useful patterns. In fact, we show some experimental results that convince us of the accuracy of our algorithms as well as their fast performances. Moreover, we are now installing our algorithms into the core of the decision tree generator in BONSAI [17], a powerful machine discovery system. We here only give basic ideas for our pruning heuristics, that are rather straightforward extensions of those developed in our previous work [8,9]. Interested readers are invited to refer to our survey report [18].
2 2.1
Finding the Best Patterns to Separate Sets of Strings Notation
Let N be the set of integers. Let Σ be a finite alphabet. An element of Σ ∗ is called a string. The length of a string w is denoted by |w|. The empty string is denoted by ε, that is, |ε| = 0. Strings x, y, and z are said to be a prefix, substring, and suffix of string w = xyz, respectively. The substring of a string w that begins at position i and ends at position j is denoted by w[i : j] for 1 ≤ i ≤ j ≤ |w|. For convenience, let w[i : j] = ε for j < i. The reversal of a string w is denoted by wR , that is, wR = w[n]w[n − 1] . . . w[1] where n = |w|. For a set S ⊆ Σ ∗ of strings, the number of strings in S is denoted by |S| and the total length of strings in S is denoted by S . Let Π = (Σ ∪ { })∗ , where is a variable length don’t care matching any string in Σ ∗ . An element q ∈ Π is a variable-length-don’t-care pattern (VLDC pattern). For example, a ab ba is a VLDC pattern with a, b ∈ Σ. We say a VLDC pattern q matches a string w if w can be obtained by replacing ’s in q with some strings. In the running example, the VLDC-pattern a ab ba matches string abababbbaa with the ’s replaced by ab, b, b and a, respectively. For any q ∈ Π, |q| denotes the sum of numbers of characters and ’s in q. 2.2
Finding the Best VLDC Patterns
We write as q u if u can be obtained by replacing ’s in q with arbitrary elements in Π. Definition 1. For a VLDC pattern q ∈ Π, we define L(q) by L(q) = {w ∈ Σ ∗ | q w}. According to the above definition, we have the following lemma. Lemma 1. For any q, u ∈ Π, if q u, then L(q) ⊇ L(u). ∗ ∗ Let good be a function from Σ ∗ × 2Σ × 2Σ to the set of real numbers. In what follows, we formulate the problem to solve. Definition 2 (Finding the best VLDC pattern according to good ). Input: Two sets S, T ⊆ Σ ∗ of strings. Output: A VLDC pattern q ∈ Π that maximizes the score of good (q, S, T ).
Discovering Best Variable-Length-Don’t-Care Patterns
89
Intuitively, the score of good (q, S, T ) expresses the “goodness” of q in the sense of distinguishing S from T . The definition of good varies with applications. For examples, the χ2 values, entropy information gain, and gini index can be used. Essentially, these statistical measures are defined by the numbers of strings that satisfy the rule specified by q. Any of the above-mentioned measures can be expressed by the following form: good (q, S, T ) = f (xq , yq , |S|, |T |), where xq = |S ∩ L(q)|, yq = |T ∩ L(q)|. When S and T are fixed, |S| and |T | are regarded as constants. On this assumption, we abbreviate the notation of the function to f (x, y) in the sequel. We say that a function f from [0, xmax ] × [0, ymax ] to real numbers is conic if – for • • – for • •
any 0 ≤ y ≤ ymax , there exists an x1 such that f (x, y) ≥ f (x , y) for any 0 ≤ x < x ≤ x1 , and f (x, y) ≤ f (x , y) for any x1 ≤ x < x ≤ xmax . any 0 ≤ x ≤ xmax , there exists a y1 such that f (x, y) ≥ f (x, y ) for any 0 ≤ y < y ≤ y1 , and f (x, y) ≤ f (x, y ) for any y1 ≤ y < y ≤ ymax .
In the sequel, we assume that f is conic and can be evaluated in constant time. The optimization problem to be tackled follows. Definition 3 (Finding the best VLDC pattern according to f ). Input: Two sets S, T ⊆ Σ ∗ of strings. Output: A VLDC pattern q ∈ Π that maximizes the score of f (xq , yq ), where xq = |S ∩ L(q)| and yq = |T ∩ L(q)|. The problem is known to be NP-hard [15], and thus we essentially have exponentially many candidates. Therefor, we reduce the number of candidates by using the pruning heuristic inspired by Morishita and Sese [16]. The following lemma derives from the conicality of function f . Lemma 2 ([8]). For any 0 ≤ x < x ≤ xmax and 0 ≤ y < y ≤ ymax , we have f (x, y) ≤ max{f (x , y ), f (x , 0), f (0, y ), f (0, 0)}. By Lemma 1 and Lemma 2, we have the next lemma, basing on which we can perform the pruning heuristic to speed up our algorithm. Lemma 3. For any two VLDC patterns q, u ∈ Π, if q u, then f (xu , yu ) ≤ max{f (xq , yq ), f (xq , 0), f (0, yq ), f (0, 0)}. 2.3
Finding the Best VLDC Patterns within a Window
We here consider a natural extension of the problem mentioned previously. We introduce an integer k called the window size. Let q ∈ Π and q[i], q[j] be the first and last characters in q that are not , respectively, where 1 ≤ i ≤ j ≤ |q|. If q matches w ∈ Σ ∗ , let w[i ], w[j ] be characters to which q[i] and q[j] can
90
S. Inenaga et al.
correspond, respectively, where 1 ≤ i ≤ j ≤ |w|. (Note that we might have more than one combination of i and j .) If there exists a pair i , j satisfying j − i < k, we say that q occurs w within a window of size k. Then the pair q, k is said to match the string w. Definition 4. For a pair q, k with q ∈ Π and k ∈ N , we define L (q, k) by L (q, k) = {w ∈ Σ ∗ | q, k matches w}. According to the above definition, we have the following lemma. Lemma 4. For any q, k and p, j with q, p ∈ Π and k, j ∈ N , if q p and j ≥ k, then L (q, k) ⊇ L (p, j). The problem to be tackled is formalized as follows. Definition 5 (Finding the best VLDC pattern and window size according to f ). Input: Two sets S, T ⊆ Σ ∗ of strings. Output: A pair q, k with q ∈ Π and k ∈ N that maximizes the score of f (xq,k , yq,k ), where xq,k = |S ∩ L (q, k)| and yq,k = |T ∩ L (q, k)|. We stress that the value of k is not given beforehand, i.e., we compute not only q but also k with which the score of function f is maximum. Therefore, the search space of this problem is Π × N , while that of the problem in Definition 3 is Π. We remark that this problem is also NP-hard. By Lemma 4 and Lemma 2, we achieve the following lemma that plays a key role for the heuristic to prune the search tree. Lemma 5. For any q, k and p, j with q, u ∈ Π and k, j ∈ N , if q u and j ≥ k, f (xp,j , yp,j ) ≤ max{f (xq,k , yq,k ), f (xq,k , 0), f (0, yq,k ), f (0, 0)}.
3
Efficient Match of VLDC Patterns
Definition 6 (Counting the matched VLDC patterns). Input: A set S ⊆ Σ ∗ of strings. Query: A VLDC pattern q ∈ Π. Answer: The cardinality of set S ∩ L(q). This is a sub-problem of the one given in Definition 3. It must be answered as fast as possible, since we are given quite many VLDC patterns as queries. Here, we utilize two practical methods which allows us to answer the problem quickly. 3.1
Using a DFA for a VLDC Pattern
Our first idea is to use a deterministic finite-state automaton (DFA) for a pattern. Given a VLDC pattern q ∈ Π, we construct a DFA that accepts L(q) and use it as a pattern matching machine (PMM) which runs over text strings in S. For any q ∈ Π, a DFA can be constructed in O(|q|) time. Lemma 6. Let S ⊆ Σ ∗ and q ∈ Π. Then |S ∩ L(q)| can be computed in O(|q|) preprocessing time and in O( S ) running time.
Discovering Best Variable-Length-Don’t-Care Patterns
91
Fig. 1. WDAWG(w) where w = abbab.
3.2
Using Wildcard Directed Acyclic Word Graphs
The second approach is to use an index structure for a text string w ∈ S that recognizes all VLDC patterns matching w. The Directed Acyclic Word Graph (DAWG) is a classical, textbook index structure [5], invented by Blumer et al. in [3]. The DAWG of a string w ∈ Σ ∗ is denoted by DAWG(w), and is known to be the smallest deterministic automaton that recognizes all suffixes of w [4]. By means of DAWG(w), we can examine whether or not a given pattern p ∈ Σ ∗ is a substring of w in O(|p|) time. Recently, we introduced Minimum All-Suffixes Directed Acyclic Word Graphs (MASDAWGs) [11]. The MASDAWG of a string w ∈ Σ ∗ , which is denoted by MASDAWG(w), is the minimization of the collection of the DAWGs for all suffixes of w. More precisely, MASDAWG(w) is the smallest automaton with |w| + 1 initial nodes, in which the directed acyclic graph induced by all reachable nodes from the k-th initial node conforms with the DAWG of the k-th suffix of w. Several important applications of MASDAWGs were given in [11], one of which corresponds to a significantly time-efficient solution to the VLDC pattern matching problem. Namely, a variant of MASDAWG(w), called Wildcard DAWG (WDAWG) of w and denoted by WDAWG(w), was introduced in [11]. WDAWG(w) is the smallest automaton that accepts all VLDC patterns matching w. WDAWG(w) with w = abbab is displayed in Fig. 1. Theorem 1. When |Σ| ≥ 2, the number of nodes of WDAWG(w) for a string w is Θ(|w|2 ). It is Θ(|w|) for a unary alphabet. Theorem 2. For any string w ∈ Σ ∗ , WDAWG(w) can be constructed in time linear in its size. For all strings in S ⊆ Σ ∗ , we construct WDAWGs. Then we obtain the following lemma that is a counterpart of Lemma 6. Lemma 7. Let S ⊆ Σ ∗ and q ∈ Π. Let N = w∈S |w|2 . Then |S ∩ L(q)| can be computed in O(N ) preprocessing time and in O(|q|·|S|) running time.
92
S. Inenaga et al.
In spite of the quadratic space requirement of WDAWGs, it is meaningful to construct them because of the following reasons. Assume that, for a string w in S, a VLDC pattern q has been recognized by WDAWG(w). We then memorize the node at which q was accepted. It allows us a rapid search of any VLDC pattern qr with r ∈ Π, since we only need to move |r| transitions from the memorized node. Therefore, WDAWGs are significantly useful especially in our situation. Moreover, WDAWGs are also helpful for pruning the search tree. Once knowing that a VLDC pattern q does not match any string in S by using the WDAWGs, we need not consider any u ∈ Π such that q u.
4
How to Compute the Best Window Size
Definition 7 (Computing the best window size according to f ). Input: Two sets S, T ⊆ Σ ∗ of strings and a VLDC pattern q ∈ Π. Output: An integer k ∈ N that maximizes the score of f (xq,k , yq,k ), where xq,k = |S ∩ L (q, k)| and yq,k = |T ∩ L (q, k)|. This is a sub-problem of the one in Definition 5, where a VLDC pattern is given beforehand. Let be the length of the longest string in S ∪ T . A short consideration reveals that, as candidates for k, we only have to consider the values from |q| up to , which results in a rather straightforward solution. In addition to that, we give a more efficient computation method, whose basic principle originates in [9]. For a string u ∈ Σ ∗ and a VLDC pattern q ∈ Π, we define the threshold value θ of q for u by θu,q = min{k ∈ N | u ∈ L (q, k)}. / L (q, k) for any k < θ If there is no such value, let θu,q = ∞. Note that u ∈ and u ∈ L (q, k) for any k ≥ θ. The set of threshold values for q ∈ Π with respect to S ⊆ Σ ∗ is defined as ΘS,q = {θu,q | u ∈ S}. A key observation is that the best window size for given S, T ⊆ Σ ∗ and a VLDC pattern q ∈ Π can be found in set ΘS,q ∪ ΘT,q without loss of generality. Thus we can restrict the search space for the best window size to ΘS,q ∪ ΘT,q . It is therefore important to quickly solve the following sub-problem. Definition 8 (Computing the minimum window size). Input: A string w ∈ Σ ∗ and a VLDC pattern q ∈ Π. Output: The threshold value θw,q . We here show our three approaches to efficiently solve the above sub-problem. The first is to adopt the standard dynamic programming method. For a string w ∈ Σ ∗ with |w| = n and a pattern q ∈ Π with |q| = m, let dij be the length of the shortest suffix of w[1 : j] that q[1 : i] matches, where 0 ≤ i ≤ m and 0 ≤ j ≤ n. We can compute all dij ’s in O(mn) time, basing on the following recurrences: d00 = 0,
Discovering Best Variable-Length-Don’t-Care Patterns
d0j = di0 =
0 ∞
if q[1] = otherwise
di−1,0 ∞
if q[1] = otherwise
93
for j ≥ 1, for i ≥ 1, and
min{di−1,j−1 +1, di,j−1 +1, di−1,j } if q[i] = di−1,j−1 +1 if q[i] = w[j] for i ≥ 1 and j ≥ 1. ∞ otherwise min1≤j≤n {dmj } if q[m] = Then θw,q = dmn otherwise. Remark that if the row dmj (1 ≤ j ≤ n) is memorized, it will save the computation time for any pattern qr with r ∈ Π. The second approach is to preprocess a given VLDC pattern q ∈ Π. We construct a DFA accepting L(q) and another DFA for L(q R ), and utilize them as PMMs running over a given string w ∈ Σ ∗ . If q[1] = (q[m] = , respectively), we have only to compute the shortest prefix (suffix, respectively) of w that q matches and return its length. We now consider the case q[1] = q[m] = . Firstly, we run the DFA for L(q) over w. Suppose that q is recognized between positions i and j in w, where 1 ≤ i < j ≤ |w| and j − i > |q|. A delicate point is that it is unsure whether w[i : j] corresponds to the shortest occurrence of q ending at position j. How can we find the shortest one? It can be found by running the DFA for L(q R ) backward, over w from position j. Assume that q R is recognized at position k, where i ≤ k < j − |q|. Then w[k : j] corresponds to the shortest occurrence of q ending at position j. After that, we resume the running of the DFA of L(q) from position k + 1, and continue the above procedure until encountering position |w|. The pair of positions of the shortest distance gives the threshold value θw,q . This method is feasible in O(m) preprocessing time and in O(mn) running time, where m = |q| and n = |w|. The third approach is to preprocess a text string w ∈ Σ ∗ , i.e., we construct WDAWG(w) and WDAWG(wR ). For any w ∈ Σ ∗ , each and every node of WDAWG(w) can be associated with a position in w [11]. Thus we can perform a procedure similar to the second approach above, which enables us to find the threshold value θw,q . This approach takes us O(n) preprocessing time and O(mn) running time, where m = |q| and n = |w|. As a result, we obtain the following: Lemma 8. Let w ∈ Σ ∗ and q ∈ Π with |w| = n and |q| = m. The threshold value θw,q can be computed in O(mn) running time. dij =
5
Computational Experiments
The algorithms were implemented in the Objective Caml Language. All calculations were performed on a Desktop PC with dual Xeon 2.2GHz CPU (though our algorithms only utilize single CPU) with 1GB of main memory running Debian Linux. In all the experiments, the entropy information gain is used as the score for which the search is conducted.
94
S. Inenaga et al. Execution time for 100 positive/100 negative completely random data (maxlen = 8)
Execution time for completely random data of length 100 (maxlen = 8)
350
350 Substring VLDC: PMM VLDC: WDAWG VLDC: WDAWG-sm VLDC in Window: PMM VLDC in Window: DP-rm
300
300
250
Time (secs)
250
Time (secs)
Substring VLDC: PMM VLDC: WDAWG VLDC: WDAWG-sm VLDC in Window: PMM VLDC in Window: DP-rm
200
150
200
150
100
100
50
50
0
0 50
100
150
200 250 300 350 Length of each string in positive/negative set
400
450
500
50
100
150
200 250 300 350 # of strings in each positive/negative set
400
450
500
Fig. 2. Execution time (in seconds) for artificial data for: different lengths of the examples (left) different number of examples in each positive/negative set (right). The maximum length of patterns to be searched for is set to 8. WDAWG-sm is matching using the WDAWG with state memoization. DP-rm is matching using the dynamic programming table with row memoization. Only one point is shown for DP-rm in the left graph, since a greater size caused memory swapping, and the computation was not likely to end in a reasonable amount of time.
5.1
Artificial Data
We first tested our algorithms on an artificial dataset. The datasets were created as follows: The alphabet was set to Σ = {a, b, c, d}. We then randomly generate strings over Σ of length l. We created 3 types of datasets: 1) a completely random set, 2) a set where a randomly chosen VLDC pattern ccd a ddad is embedded in the positive examples, and 3) a set where a pair of a VLDC pattern and a window size ccd a ddad , 19 is embedded in the positive examples. In 2) and 3), a randomly generated string is used as a positive example if the pattern matches it, and used as a negative example otherwise, until both positive and negative set sizes are n. Examples for which the set size exceeds n are discarded. Fig. 2 shows the execution times for different l and n, for the completely random dataset. We can see that the execution time grows linearly in n and l as expected, although the effect of pruning seems to take over for VLDC patterns in the left graph, when the length of each sequence is long. Searching for VLDC patterns and window sizes using dynamic programming with row memoization, does not perform very well. Fig. 3 shows the execution times for different maximum lengths of VLDC patterns to look for, for the 3 datasets (The length of a VLDC pattern is defined as the length of the pattern representation, excluding any ’s on the ends). We can see that the execution time grows exponentially as we increase the maximum pattern length searched for, until the pruning takes effect. The lower left graph in Fig. 3 shows the effect of performance of an exhaustive search, run on the completely random dataset, compared to searches with the branch and bound pruning for the different datasets. The pruning is more effective when it is more likely to have a good solution.
Discovering Best Variable-Length-Don’t-Care Patterns
95
Fig. 3. Execution time (in seconds) for artificial data for different maximum lengths of patterns to be searched for with: completely random data (upper left), VLDC and window size embedded data (upper right), VLDC embedded data (lower left). The lower right graph shows the effect of pruning of the search space for the different data sets, compared to exhaustive search on the completely random dataset.
5.2
Real Data
To show the usefulness of VLDC patterns and windows, we also tested our algorithms on actual protein sequences. We use the data available at http://www.cbs.dtu.dk/services/TargetP/, which consists of protein sequences which are known to contain protein sorting signals, that is, (in many cases) a short amino acid sequence segment which holds the information which enables the protein to be carried to specified compartments inside the cell. The dataset for plant proteins consisted of: 269 sequences with signal peptide (SP), 368 sequences with mitocondrial targeting peptide (mTP), 141 sequences with chloroplast transit peptide (cTP), and 162 “Other” sequences. The average length of the sequences was around 419, and the alphabet is the set of 20 amino acids. Using the signal peptides as positive examples, and all others as negative examples, we searched for the best pair p, k with maximum length of 10 using PMMs. To limit the alphabet size, we classify the amino acids into 3 classes {0, 1, 2}, according to the hydropathy index [13]. The most hydrophobic amino acids {A, M, C, F, L, V, I} (hydropathy ≥ 0.0) are converted to 0, {P,Y,W,S,T,G} (−3.0 ≤ hydropathy < 0.0 ) to 1, and {R, K, D, E, N, Q, H}
96
S. Inenaga et al.
(hydropathy < −3.0 ) to 2. We obtained the pair 0 00 00000 , 26, which occurs in 213/269 = 79.2% of the sequences with SP, and 26/671 = 3.9% of the other sequences. The calculation took exactly 50 minutes. This pattern can be interpreted as capturing the well known hydrophobic h-region of SP [22]. Also, the VLDC pattern suggests that the match occurs in the first 26 amino acid residues of the protein, which is natural since SP, mTP, cTP are known to be Nterminal sorting signals, that is, they are known to appear near the head of the protein sequence. A best substring search quickly finds the pattern 00000001 in 36 seconds, but only gives us a classifier that matches 152/269 = 56.51% of the SP sequences, and 41/671 = 6.11% of the others. For another example, we use the mTP set as positive examples, and the SP and Other sets as negative examples. This time, we convert the alphabet according to the net charge of the amino acid. Amino acids {D, E} (negative charge) are converted to 0, {K, R} (positive charge) to 1, and the rest {A, L, N, M, F, C, P, Q, S, T, G, W, H, Y, I, V} to 2. The calculation took about 21 minutes and we obtain the pair 2 1 1 2221 , 28 which occurs in 334/368 = 90.76% of the mTP sequences and (73/431 = 16.94%) of the SP and Other sequences. This pattern can also be regarded as capturing existing knowledge about mTPs [23]: They are fairly abundant in K or R, but do not contain much D or E. The pattern also suggests a periodic appearance of K or R, which is a characteristic of an amphiphilic α-helix that mTPs are reported to have. A best substring search finds pattern 212221 in 20 seconds, which gives us a classifier that matches 318/368 = 86.41% of sequences with mTP and 255/431 = 59.16% of the other sequences.
References 1. D. Angluin. Finding patterns common to a set of strings. J. Comput. Sys. Sci., 21:46–62, 1980. 2. R. A. Baeza-Yates. Searching subsequences (note). Theoretical Computer Science, 78(2):363–376, Jan. 1991. 3. A. Blumer, J. Blumer, D. Haussler, A. Ehrenfeucht, M. T. Chen, and J. Seiferas. The smallest automaton recognizing the subwords of a text. Theoretical Computer Science, 40:31–55, 1985. 4. M. Crochemore. Transducers and repetitions. Theoretical Computer Science, 45:63–86, 1986. 5. M. Crochemore and W. Rytter. Text Algorithms. Oxford University Press, New York, 1994. 6. G. Das, R. Fleischer, L. Gasieniec, D. Gunopulos, and J. K¨ arkk¨ ainen. Episode matching. In Proc. 8th Annual Symposium on Combinatorial Pattern Matching (CPM’97), volume 1264 of Lecture Notes in Computer Science, pages 12–27. Springer-Verlag, 1997. 7. D. Gusfield. Algorithms on Strings, Trees, and Sequences. Cambridge University Press, New York, 1997. 8. M. Hirao, H. Hoshino, A. Shinohara, M. Takeda, and S. Arikawa. A practical algorithm to find the best subsequence patterns. In Proc. The Third International Conference on Discovery Science, volume 1967 of Lecture Notes in Artificial Intelligence, pages 141–154. Springer-Verlag, 2000.
Discovering Best Variable-Length-Don’t-Care Patterns
97
9. M. Hirao, S. Inenaga, A. Shinohara, M. Takeda, and S. Arikawa. A practical algorithm to find the best episode patterns. In Proc. The Fourth International Conference on Discovery Science, volume 2226 of Lecture Notes in Artificial Intelligence, pages 435–440. Springer-Verlag, 2001. 10. S. Inenaga, A. Shinohara, M. Takeda, H. Bannai, and S. Arikawa. Space-economical construction of index structures for all suffixes of a string. In Proc. 27th International Symposium on Mathematical Foundations of Computer Science (MFCS’02), Lecture Notes in Computer Science. Springer-Verlag, 2002. To appear. 11. S. Inenaga, M. Takeda, A. Shinohara, H. Hoshino, and S. Arikawa. The minimum dawg for all suffixes of a string and its applications. In Proc. 13th Annual Symposium on Combinatorial Pattern Matching (CPM’02), volume 2373 of Lecture Notes in Computer Science, pages 153–167. Springer-Verlag, 2002. 12. S. R. Kosaraju. Fast pattern matching in trees. In Proc. 30th IEEE Symp. on Foundations of Computer Science, pages 178–183, 1989. 13. J. Kyte and R. Doolittle. A simple method for displaying the hydropathic character of a protein. J. Mol. Biol., 157:105–132, 1982. 14. H. Mannila, H. Toivonen, and A. I. Verkamo. Discovering frequent episode in sequences. In Proc. 1st International Conference on Knowledge Discovery and Data Mining, pages 210–215. AAAI Press, 1995. 15. S. Miyano, A. Shinohara, and T. Shinohara. Polynomial-time learning of elementary formal systems. New Generation Computing, 18:217–242, 2000. 16. S. Morishita and J. Sese. Traversing itemset lattices with statistical metric pruning. In Proc. of the 19th ACM SIGACT-SIGMOD-SIGART Symposium on Principles of Database Systems, pages 226–236. ACM Press, 2000. 17. S. Shimozono, A. Shinohara, T. Shinohara, S. Miyano, S. Kuhara, and S. Arikawa. Knowledge acquisition from amino acid sequences by machine learning system BONSAI. Transactions of Information Processing Society of Japan, 35(10):2009– 2018, 1994. 18. A. Shinohara, M. Takeda, S. Arikawa, M. Hirao, H. Hoshino, and S. Inenaga. Finding best patterns practically. In Progress in Discovery Science, volume 2281 of Lecture Notes in Artificial Intelligence, pages 307–317. Springer-Verlag, 2002. 19. T. Shinohara. Polynomial-time inference of pattern languages and its applications. In Proc. 7th IBM Symp. Math. Found. Comp. Sci., pages 191–209, 1982. 20. Z. Tron´ıˇcek. Episode matching. In Proc. 12th Annual Symposium on Combinatorial Pattern Matching (CPM’01), volume 2089 of Lecture Notes in Computer Science, pages 143–146. Springer-Verlag, 2001. 21. E. Ukkonen. On-line construction of suffix trees. Algorithmica, 14(3):249–260, 1995. 22. G. von Heijne. The signal peptide. J. Membr. Biol., 115:195–201, 1990. 23. G. von Heijne, J. Steppuhn, and R. G. Herrmann. Domain structure of mitochondrial and chloroplast targeting peptides. Eur. J. Biochem., 180:535–545, 1989.
A Study on the Effect of Class Distribution Using Cost-Sensitive Learning Kai Ming Ting Gippsland School of Computing and Information Technology, Monash University, Victoria 3842, Australia. [email protected]
Abstract. This paper investigates the effect of class distribution on the predictive performance of classification models using cost-sensitive learning, rather than the sampling approach employed previously by a similar study. The predictive performance is measured using the cost space representation, which is a dual to the ROC representation. This study shows that distributions which range between the natural distribution and the balanced distribution can also produce the best models, contrary to the finding of the previous study. In addition, we find that the best models are larger in size than those trained using the natural distribution. We also show two different ways to achieve the same effect of the corrected probability estimates proposed by the previous study.
1
Introduction
Since the recent revelation of the fact that the best performing model is not obtained from the training data with matching class distribution as that in the test data, it has been a growing interest in investigating the effect of class distribution on classifier performance. One such recent study is conducted by Weiss & Provost [11] using a sampling approach. Given a test set of a fixed class distribution, the study investigates the conditions under which the best classifiers can be trained from different class distributions. This study has the same aim as the previous study but differs in two important aspects. First, we use the cost-sensitive learning instead of the sampling approach. Though the two methods are equivalent conceptually, cost-sensitive learning maintains the same training set while changing the class distribution; but the sampling approach actually adds or reduces the number of training examples in each class in order to maintain the same training size. As a result, each class distribution does not have exactly the same content as the others, and a substantial amount of available (majority class) data is not used in the highly skewed distribution data sets. None of these problems appear in our approach. We identify these problems as the limiting factor in the previous finding (described later in Section 3.1). Furthermore, the task of determining the best training class distribution(s) also becomes simpler using cost-sensitive learning approach because only a learning parameter value need to be altered. Second, we conduct a finer analysis using the cost space representation, in addition to S. Lange, K. Satoh, and C.H. Smith (Eds.): DS 2002, LNCS 2534, pp. 98–112, 2002. c Springer-Verlag Berlin Heidelberg 2002
A Study on the Effect of Class Distribution Using Cost-Sensitive Learning
99
using a similar coarse performance measure such as the area under ROC curves (AUC) used in the previous study. This paper examines the previous finding [11] that (i) a model that trained from the balanced distribution will perform no worse or better than that trained from the natural distribution, and (ii) the best distribution to train a model is generally the one that biases the minority class. We show in this paper that this is only true under certain condition. On a finer analysis, we show that the finding should be revised to include distributions that range between the natural distribution and the balanced distribution. Using more general performance measures than predictive accuracy, we affirm a previous result [10] which shows that a larger size tree performs better than a tree trained from a commonly used induction method. Unlike the previous work which uses a different induction method, we show that one can induce a larger size and better performing tree using exactly the same induction method. Weiss & Provost [11] show that an improved performance in terms of predictive accuracy can be obtained by correcting the probability of each leaf in a decision tree under changed operating conditions. We show in this paper that an equivalent outcome can be obtained by modifying the decision rule instead, leaving the induced model unchanged. We show two ways of achieving that, depending on whether the changed condition is expressed as probability or cost. We begin the next section by describing the cost space representation pertinents to this work. Section 3 reports the experiments and results of examining the utility of the balanced distribution, and whether better performing trees are larger in size. Section 4 gives an analysis for an alternative to the corrected probability. We present a discussion and a summary in the last two sections.
2
Using Cost Curves to Measure Performance
Drummond & Holte [3] introduce the cost space representation in which the expected cost of a classification model is represented explicitly. It is a dual to the ROC representation but has advantages over the latter when a comparison of model performance in terms of the expected cost is required. Let p(a) be the probability (prior) of a given example being in class a, and C(a|b) be the cost of misclassifying a class b example as being in class a. The normalised expected cost expressed in terms of true positive rate (T P ), false positive rate (F P ) and probability-cost function (P CF ) is defined [3] as N E[C] = (1 − T P )P CF (+) + F P (1 − P CF (+)) = (1 − T P − F P )P CF (+) + F P, where
positives correctly classif ied , total positives negatives incorrectly classif ied FP = , and total negatives p(+)C(−|+) P CF (+) = . p(+)C(−|+) + p(−)C(+|−) TP =
(1) (2)
100
K.M. Ting
Fig. 1. Cost lines and curves: Part (a) shows a cost line produced using a fixed decision threshold, and part (b) shows the cost lines produced using different decision thresholds from the same model. The order in the legends corresponds to the anti-clockwise order starting from the line on the right. The two diagonal cost lines that intercept at point (0.5, 0.5) are the performance of the default classifiers that always predict + or −, respectively. The cost curve for the model is the lowest envelop of all the cost lines for the model, indicated as the dotted line. Note that not all the cost lines are shown. In our implementation, each curve is sampled at 0.01 interval along P CF (+)-axis.
The performance of a classification model that uses a fixed decision threshold is represented by a pair {T P , F P }. Given the pair, it can be represented as a line in the cost space, which consists of the normalised expected cost in the yaxis and P CF (+) in the x-axis, indicated by the linear equation (2). Because both are normalised, they range from 0 to 1. An example of a cost line is shown in Figure 1(a). By varying the decision threshold1 of the same model, different cost lines are obtained (as shown in Figure 1(b)). The cost curve representing the performance of the model that uses varying decision thresholds is the lowest envelop of all cost lines produced by the model. Provost & Fawcett [6] shows an algorithm where one can obtain all pairs of {T P , F P } in one pass for all the different thresholds of a model. P CF (+) denotes the operating condition when one wants to use a model for testing. The area under the cost curve is the total expected cost. The difference in area under two curves is a coarse measure of the expected advantage of a model 1
A decision threshold is the cut-off level used to decide the final prediction of a classification model. In a two-class problem, the final prediction is class positive if the model’s posterior probability of a test example is above the threshold; otherwise it is class negative. When the threshold is changed, the model’s performance also changes.
A Study on the Effect of Class Distribution Using Cost-Sensitive Learning
101
over another. A model which has the lower normalised expected cost under all operating conditions is said to strictly dominate the other. We are interested in using unbalanced distribution data in this study as such data occurs in many real-world applications. We will denote the minority class as + and the majority class as −. The cost ratio (M ) is defined as the ratio of the cost of misclassifying the minority class and the cost of misclassifying the majority class, that is C(−|+)/C(+|−). When the cost ratio is unity, the normalised expected cost reduces to error rate and P CF (+) to p(+). While the cost ratio reflects the condition for the given data, one can use this information to influence the model induction process in training. This is what we do by using M as a training parameter in cost-sensitive learning2 . Rather than using the entire area under the cost curve (AUC) as a measure of performance we use a variant, AUC*, as a coarse measure for performance3 . AUC* is defined as follows. For M > 1, AUC* is the part of the area under the cost curve where P CF (+) ranges between the training prior and 1.0, assuming the changed operating conditions are solely due to changes in cost and the prior is unchanged. This is equivalent to a changed operating condition solely due to changes in prior and the cost is unchanged. For M < 1, AUC* is the other part of the area where P CF (+) ranges between 0.0 and the training prior. We only use the first definition since we are interested in unbalanced data. A finer analysis of model performance requires to identify the operating range in which a model is better than the other. In this case, the normalised expected cost is used instead. We use both measures here. Note that all AUC* values shown in this paper must ×10−2 to obtain the actual values.
3
Experiments
We use two sets of data sets obtained from the UCI repository [1]. In the first set where each data set has the total data size larger than 4000, we split each set into training and testing sets. The large data size allows us to produce learning curves, where each point in the curve is an average over multiple runs using a subset of the training set. In other cases, a single run is conducted for each data set. The main reason is that only a cost curve produced from a single run allows one to identify the threshold of the model for certain operating condition. An average cost curve produced from multiple runs does not allow one to do that. The second set of data sets has size less than 4000. Here we perform 2
3
An instance weighting approach to changing class distribution in decision tree induction has been used by a number of researchers. A description of the approach can be found in [8], which is the approach used in this paper. Refer to [9] for a discussion on the factors affecting the selection of either AUC or AUC*. AUC* is preferred for the method of generating cost curves we used here— from a single model with varying decision thresholds. Also, the difference between AUC and AUC* is a constant for unbalanced data because the model induced using M < 1 is likely to be the default model. The constant approaches zero as the data becomes highly imbalance.
102
K.M. Ting
Table 1. Description of data sets, evaluation methods and M values for balanced distribution. Data set D01. D02. D03. D04. D05. D06. D07. D08. D09. D10. D11. D12. D13.
% Minority Size and Method M for Prior Train Test Balanced Distribution Coding 50.0 15000 5000 1.00 Abalone 31.7 2784 1393 2.15 Adult 24.1 32561 16281 3.15 Satellite 9.7 4290 2145 9.29 Pendigits 9.6 7494 3498 9.42 Letter-a 3.9 13333 6667 24.64 Nursery 2.6 8640 4320 38.22 Nettalk-s 1.1 3626 1812 89.91 Kr-vs-kp 47.8 10CV of 3196 1.09 German 30.0 10CV of 1000 2.33 Splice 24.0 10CV of 3175 3.17 Solar Flare 15.7 10CV of 1389 5.37 Hypothyroid 4.8 10CV of 3168 19.83
a stratified 10-fold cross-validation to produce an average cost curve in order to get a good estimate. Both sets have data sets with percentage of minority class spanning from 50% to about 1%, that is from balanced to highly skewed distributions. The description of the data sets and the evaluation methods are listed in Table 1. There are seven data sets which have more than two classes and they are converted to two classes (marked with ). In all splits or partitions, the class distribution is maintained. Only two-class problems are considered because of the cost space representation’s existing limitation to this class of problems [3]. The last column of Table 1 shows the value of training parameter M used in order to train a model from balanced distribution in each data set. The Appendix shows how this is obtained. We use the decision tree induction algorithm C4.5 [7] and its default setting in all experiments, while taking the following modifications into consideration. The algorithm is modified to take cost ratio into consideration in the training process. For example, with M =2, every minority class training instance will be weighted twice as high as every majority class instance during the tree induction process [8]. Cost-sensitive pruning is done in accordance with the modified weights, though the default pruning method of C4.5 is used unaltered. The Laplace estimate is used to compute the class posterior probability for a leaf of the tree. This allow us to compute all pairs of {T P , F P } for a test set in one pass using the algorithm of Provost & Fawcett [6]. We report the results of pruned trees using the gain ratio criterion4 4
Despite previous claims of better performance using unpruned trees [2] and using the DKM splitting criterion [4], a recent study [9] shows that these claims are a result of using sub optimal models for comparison. When optimal models are used, the outcomes are different.
A Study on the Effect of Class Distribution Using Cost-Sensitive Learning
103
In the following experiments in each data set, a set of selected M values are explored. We use the term “the best” to refer to the best among those produced in the set of the selected M values.
Fig. 2. Learning curves for models trained using different M values. Each curve is a result of training using one M value at different training sizes. Insignificant curves are omitted to enhance readability.
3.1
Trees Trained from Balanced Distribution
In Weiss & Provost’s [11] experiments, they find that “..the optimal distribution generally contains between 50% and 90% minority-class examples. In this situation, the strategy of always allocating half of the training examples to the minority class, while it will not always yield optimal results, will generally lead
104
K.M. Ting
to results which are no worse than, and often superior to, those which use the natural class distribution.” We examine their proposition in this section. In terms of AUC*, a learning curve is produced for each M value. The base tree is the one trained with M =1, i.e., natural distribution. Trees of M =2,5,10,20,50,100,1000 and a value corresponding to the balanced distribution (shown in the last column of Table 1) are trained. Each learning curve is produced from 10% to 100% of the entire training set, at 10% intervals. The class distribution is maintained in all cases and the high percentage data always includes the low percentage data. The same testing set is used for all runs in each data set. Each point in the curve is an average over 10 runs, except at 100% where only one run is carried out. The learning curves are shown in Figure 2.
Fig. 3. Learning curves for models trained using different M values. (continue)
A Study on the Effect of Class Distribution Using Cost-Sensitive Learning
105
Observation of the learning curves shows that trees trained from the natural distribution almost always perform worse than a lot of trees trained using a changed distribution biased towards minority class. The only exception is the coding data set in small training sizes, whose natural distribution is balanced. Balanced class distribution appears to be the best (or close to the best) to use when the training size is small. This happens in eight data sets: ≤ 20% in nettalk-stress, ≤ 30% in coding and satellite, ≤ 40% in letter-a, ≤ 60% in pendigits, ≤ 70% in nursery, ≤ 80% in adult, and ≤ 90% in abalone (note that abalone has training size < 3000). For a large training size, usually a M value larger than the one corresponding to the balanced distribution is the best. However, there are some exceptions. First, the natural distribution is better than the balanced distribution in some high percentage training data in satellite, pendigits, and adult. Second, a tree trained with a value of M which is less than that of the balanced distribution can sometimes be better than a tree trained with the balanced distribution. Examples can be found in some cases in adult, pendigits, and nettalk-stress. Overall, this result does not contradict with Weiss & Provost’s finding as their result is equivalent to one low to medium percentage point in our learning curve. Their training size is much smaller than ours because they use sampling; thus, it is restricted by the total number of minority class instances to vary the percentage of minority class from 2% to 95% of the training data in their experimental design.
Fig. 4. Part (a) shows five cost curves, each representing the performance of a model induced using one value of M ; Part (b) shows the same curves plotted with cost ratio as x-axis, instead of P CF (+). Note that only part of the curves are shown in part (b) where cost ratio ranges 1–20. In this case, the best operating range is 1.0–1.5 for the model built using M = 1 and 1.6–100 for the model built using M = 2.
106
K.M. Ting
Table 2. The best operating range, in terms of cost ratios, obtained from the trees trained using M =1,2,5,10,20,50,100,1000 and a value corresponding to the balanced class distribution. In kr-vs-kp, M =1 and M =1.09 produce the same tree. The marker indicates the relative position of the M value corresponding to the balanced distribution for each data set, shown in Table 1.
D01 D02 D03 D04 D05 D06 D07 D08 D09 D10 D11 D12 D13
3.2
M=1 1.0-1.5 1.0-1.5 150-200 1-30 1.6-12 1.0-2.0
The best operating range of cost ratios in testing for each M used in training M=2 M=5 M=10 M=20 M=50 M=100 M=1000 1.6-100 1.0-2.2 2.3-3.7 3.8-11 12-100 1.6-3.5 3.6-8.9 9-22 23-55 56-300 1-6.9 7-15 16-300 1-300 210-300 16-140 210-300 6.7-200 31-100 1.0-2.3 2.4-16 1.0-1.5 13-300 1.0-1.7 5.3-11 12-250 2.1-300
M=Bal. 1.0-1.5
1-15 4.7-200 1-30 17-230 1.8-5.2
Finer Analysis Using Cost Curves
For a finer analysis using the normalised expected cost, cost curves such as those in Figure 4(a) are produced and the best operating range to be used for testing in each of the thirteen data sets are recorded. The result is shown in Table 2. This is equivalent to getting the lowest envelop of all cost curves from the trees trained using M =1,2,5,10,20,50,100,1000, and a value corresponding to the balanced distribution. For ease of reference, the operating range is converted to cost ratio from P CF (+). The Appendix shows the relationship between cost ratios and priors in terms of P CF (+). Figure 4(b) show example cost curves for D01:coding data set plotted using cost ratio in the x-axis, in which the result in Table 2 is extracted from. Weiss & Provost’s finding quoted in the last section can be translated to the following: The best operating range in each data set shall lie on the righthand side of the balanced distribution (marked by ) in Table 2. In contrast, we observe that • A substantial portion of the range in many data sets lies on the left-hand side of the marker. • In two unbalanced data sets (D05:pendigits and D13:hypothyroid), the entire range lies on the left-hand side. • Only in three data sets (D04:satellite, D07:nursery and D08:nettalk-stress), the entire range lies on the right-hand side. This is disregarding the range where the default classifiers are the best. Note that the best ranges in nursery and nettalk-stress do not begin with 1.0, because the best classifier in those ranges is the default classifier. This result reveals a fact that in highly skewed
A Study on the Effect of Class Distribution Using Cost-Sensitive Learning
107
distribution data, we are usually not interested in operating at P CF (+) corresponding to the natural distribution (or any values close to it) because the default classifier turns out to be the best. The default model is also the best model for cost ratio > 200 in nettalk-stress. The two naturally balanced distribution data sets where the best ranges must lie on the right-hand side are not counted. • Trees trained using the changed balanced distribution contribute to produce the best performance in some range in four data sets only (indicated in the last column in Table 2), excluding the two naturally balanced distribution data sets. Note that only a few trees are required to cover a wide range of cost ratios. One extreme example is in pendigits where a tree trained with M =5 is the best for the entire range. Other examples are in D01:coding, D08:nettalk, D09:kr-vs-kp and D13:hypothyroid—except for the cost ratios close to the natural distribution, only one tree is required for the rest of the range. 3.3
Are Better Performing Trees Larger in Size?
Webb [10] introduces a method of grafting trees that adds nodes to an existing tree and the larger trees perform better in terms of accuracy. Using more general measures such as AUC* and the expected cost, this section investigates whether a tree that performs better than the one trained from the natural distribution has larger size. The first type of better performing trees are those trained from the M value with the best AUC*. The second type of better performing trees are those trained from the best M for the natural distribution, in terms of the expected cost. Table 3 shows the tree sizes for trees trained from the natural distribution, and the two types of better performing trees. The M value used to get the best AUC* is shown in the second column. The best M used to train the best performing tree for the natural distribution is shown in the last column. We measure tree size as the total number of leaves predicting either the minority class or the majority class. The trees trained with a M value that produce the best AUC* for imbalance data sets have tree sizes larger than those trained using the natural distribution in eight data sets. In kr-vs-kp and splice, the best trees are trained from the natural distribution (see the second column in Table 3). The only real exceptions are coding, satellite and nursery. The trees trained with the best M have sizes larger than those trained with M =1 in six data sets. Because there are four data sets in which the best M is 1 and the best tree in nursery and nettalk-stress is the default classifier, the only real exception is satellite. This result points to an alternative method other than the tree grafting method [10] that produces a larger tree than those trained using the intended cost ratio but performs better. Instead of using a different induction algorithm, the method is to find the best M and train a tree from it using exactly the same induction algorithm.
108
K.M. Ting
Table 3. Leaf sizes for trees trained from the natural distribution, from the M value with the best AUC*, and from the best M for the natural distribution. Sa and Si are the total numbers of leaves predicting majority class and minority class, respectively. Boldface indicates that the sizes are larger than those in M =1. Note that though a nontrivial classifier is induced using M =1 in nursery, the default classifier is still the best to operate on a point corresponding to M =1 or the natural distribution. indicates that the default models are used. Data set
(M)
M=1
Sa/Si Coding 2 804/766 Abalone 5 18/15 Adult 10 285/241 Satellite 20 61/46 Pendigits 5 21/19 Letter-a 50 22/10 Nursery 100 38/23 Nettalk-s 1000 1/0 Kr-vs-kp 1 9.5/15.9 German 5 26.5/25.0 Splice 1 23.8/8.4 S.Flare 10 1.8/0.8 Hypo 10 4.3/1.5
4
(M) with Best M Best M Best for AUC* nat.dist. Sa /Si Sa/Si 563/524 804/766 1 22/21 30/25 2 539/628 285/241 1 51/43 51/43 20 22/20 22/20 5 29/19 29/19 24.64 25/18 1/0 132/77 1/0 9.5/15.9 9.5/15.9 1 27.5/29.6 35.7/35.1 2 23.8/8.4 24.7/15.6 5 9.0/15.7 10.5/10.5 2 11.1/7.1 4.3/1.5 1
An Alternative to Corrected Probability: Modifying the Decision Rule
Weiss & Provost [11] show that in order to achieve better predictive accuracy, the estimated probability at a leaf of a decision tree shall be corrected in the following form: N+ p(+) p (+) : , , where k = N+ + kN− p(−) p (−)
(3)
and Na is the number of class a training examples at the leaf, and p(a) and p (a) are the prior probabilities for class a in the training set and testing set, respectively. Instead of correcting the probability at each leaf of a tree, we show here that one can modify the decision rule instead, leaving the induced tree unchanged. There are two ways to do this, depending on whether the changed condition is expressed as (i) probability or (ii) misclassification cost. + Let P = P (+|x) = N+N+N be the uncorrected probability for class + given − an example x. (i) For changed probability, equation (3) can be re-written by dividing the numerator and denominator by N+ + N− , and expressed in terms of P and k as follows.
A Study on the Effect of Class Distribution Using Cost-Sensitive Learning
109
P P + k(1 − P ) The tree that uses the following decision rule to make a prediction achieves the same effect of equation (3) which uses the fixed decision threshold of 0.5. P redict class + if P : k(1 − P ) > 1; predict class − otherwise.
(4)
(ii) If the changed condition is expressed as misclassification cost, we show here that using the uncorrected probability and the minimum expected cost (MEC) criterion to make a decision [5,8] can achieve the same effect. Let E(a|x) be the expected cost of predicting class a, given an example x. The MEC criterion for two-class problems is defined as follows. P redict class + if E(−|x) : E(+|x) > 1; predict class − otherwise. (5) Where E(−|x) = P (+|x)C (−|+) = P C (−|+). E(+|x) = P (−|x)C (+|−) = (1 − P )C (+|−).
(6) (7)
Recall C(a|b) is the cost of misclassifying a class b example as class a, during training; and C (a|b) is the changed cost during testing. Let T P and F P be the true positive rate and the false positive rate. The slope of the line connecting two points {T P1 , F P1 } and {T P2 , F P2 } in the ROC representation is defined as follows [6]. p(−)C(+|−) T P2 − T P 1 . = F P2 − F P1 p(+)C(−|+) Let p be the changed prior with fixed C; and C be the equivalent changed misclassification cost with fixed p, such that: p(+)C (−|+) p (+)C(−|+) = . p (−)C(+|−) p(−)C (+|−) Using the relation k in equation (3), C (−|+) C(−|+) p (+) p(+) 1 = = . / / C (+|−) C(+|−) p (−) p(−) k C (−|+) 1 C(−|+) = . C (+|−) k C(+|−) Then, substitute equations (6) and (7) into (5) and replace equivalent as in equation (8) give C (−|+) P , (1 − P ) C (+|−) 1 C(−|+) P = (1 − P ) k C(+|−)
E(−|x) : E(+|x) =
(8) C (−|+) C (+|−)
with its
(9) (10)
110
K.M. Ting
Since C(−|+) C(+|−) is assumed to be unity in Weiss & Provost’s formulation, substitute equation (10) into equations (5) reveals that the prediction using the MEC criterion (with the uncorrected probability) is equivalent to that using equation (4); and thus equivalent to that using the corrected probability with the fixed decision threshold of 0.5.
5
Discussion
It is important to point out that there is a subtle difference between the previous work [11] and this work. Weiss & Provost study the effect of class distribution with a different information content in each class distribution; whereas our work studies the effect with exactly the same information content in each class distribution. Though both aim at investigating the conditions under which the best classifiers can be trained from different class distribution, the drivers behind them differ. The former study is driven by the following need: To purchase a data set with the optimal class distribution, under the constraint of a limited fund. Assuming there is a large pool of data to purchase from. This study is driven to find an optimal cost (or ROC) curve from a given data set that allows one to use the optimal models under different operating conditions. If one has to first purchase a data set from a large data pool and then build an optimal cost curve, the results of the two studies are complementary which contribute to the following recommendation—purchase a balanced class distribution data set (guided by the result of the former study) and then construct the optimal cost curve using cost-sensitive learning.
6
Summary
We reveal the limitation of the sampling approach imposed on the experimental design which constrains the finding of the previous research to small data size only. Based on the similar coarse measure as used by the previous study, both studies produce the same finding for small training size. However, there are some discrepancies for large training size. On a finer analysis using cost curves, the discrepancies become apparent that they lead us to conclude: To train the best model, one cannot ignore the M values which lie between unity (i.e., the natural distribution) and those less than that corresponding to the balanced distribution. Using the general measures of the normalised expected cost and AUC*, our result in Section 3.3 shows that a larger size tree performs better than a tree trained from the natural distribution, in which both trees are trained from the same induction algorithm. While this affirms the previous result, the previous work was using a different induction algorithm to achieve that. The current work contributes to open an avenue to find a larger and better performing tree using exactly the same induction algorithm. We show that the corrected probability can be better implemented using a modified decision rule, leaving the induced model unchanged. Two ways of
A Study on the Effect of Class Distribution Using Cost-Sensitive Learning
111
modifying the decision rule are shown to give predictive decision equivalent to that given by the corrected probability. The ways differ in whether the changed operating condition is expressed as probability or misclassification cost. The revelation mentioned at the beginning of this paper is implied throughout most parts of this paper without further discussion: A trained model does not operate optimally on the data drawn from the same distribution as the training data. The reader is referred to [9] for a discussion of this issue. Acknowledgements. Part of this work was completed when the author was visiting Osaka Univeristy, supported by a fellowship provided by Japan Society for the Promotion of Science. Ross Quinlan provides C4.5. Foster Provost and Gary Weiss point out that the results from their study and this work can be complementary. Comments from the anonymous reviewers have helped to improve this paper.
References 1. Blake, C. & Merz, C.J. UCI Repository of machine learning databases. [http://www.ics.uci.edu/˜mlearn/MLRepository.html]. Irvine, CA: University of California (1998). 2. Bradford, J., Kunz, C., Kohavi, R., Brunk, C., & Brodley, C. Pruning decision trees with misclassification costs. Proceedings of the European Conference on Machine Learning. (1998) 131–136. 3. Drummond C. & Holte R. Explicitly Representing Expected Cost: An Alternative to ROC Representation. Proceedings of the Sixth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (2000) 198–207. 4. Drummond C. & Holte R. Exploiting the Cost (In)sensitivity of Decision Tree Splitting Criteria. Proceedings of The Seventeenth International Conference on Machine Learning. San Francisco: Morgan Kaufmann. (2000) 239–246. 5. Michie, D., D.J. Spiegelhalter, & C.C. Taylor. Machine Learning, Neural and Statistical Classification. Ellis Horwood Limited. (1994). 6. Provost, F. & Fawcett, T. Robust Classification for Imprecise Environments. Machine Learning 42 (2001) 203–231. 7. Quinlan, J.R. C4.5: Program for Machine Learning. Morgan Kaufmann. (1993). 8. Ting, K.M. An Instance-Weighting Method to Induce Cost-Sensitive Trees. IEEE Transactions on Knowledge and Data Engineering. Vol. 14, No. 3. (2002) 659–665. 9. Ting, K.M. Issues in Classifier Evaluation using Optimal Cost Curves. Proceedings of The Nineteenth International Conference on Machine Learning. San Francisco: Morgan Kaufmann. (2002) 642–649. 10. Webb, G. Decision tree grafting from the all-tests-but-one partition. Proceedings of the 16th International Joint Conference on Artificial Intelligence. San Fransisco: Morgan Kaufmann. (1999) 702–707. 11. Weiss, G. & Provost, F. The Effect of Class Distribution on Classifier Learning: An Empirical Study. Technical Report ML-TR-44, Department of Computer Science, Rutgers University. (2001).
112
K.M. Ting
Appendix: The Relationship between P CF and Cost Ratio Operating conditions can be expressed as either P CF (or its components) or cost ratios. The original cost curves use P CF to denote the operating conditions [3]. However, there is an advantage to plot cost curves using cost ratios as xaxis, instead of P CF . Because a unity cost ratio corresponds to the natural distribution and it does not change from one data set to another, unlike P CF , this allows easy identification of the natural distribution operating condition in the cost curves. On the other hand, P CF (+) = 0.5 always indicates the balanced distribution. The relationship between cost ratio and P CF (+), including those in these two special conditions are given below. Let p and C denote the prior and misclassification cost of an operating condition, respectively. To avoid confusion with cost ratio M as the learning parameter for an induction algorithm, we use cost ratio R to denote the operating condition. By substituting R = C(−|+)/C(+|−) into the definition of P CF from [3]: p(+)C(−|+) . p(+)C(−|+) + p(−)C(+|−) p(+)R . = p(+)R + p(−) P CF (+)p(−) = p(+)R(1 − P CF (+)). P CF (+) p(−) R= . 1 − P CF (+) p(+) P CF (+) =
Thus, R corresponds to a changed probability by the factor special values are given below.
P CF (+) 1−P CF (+) .
Three
R = 1.0 when P CF (+) = p(+). p(−) when P CF (+) = 0.5. R= p(+) R = ∞ when P CF (+) = 1.0. R = 1.0 corresponds to the operating condition that matches the natural distribution. R = p(−) p(+) or P CF (+) = 0.5 corresponds to the operating condition which is equivalent to the balanced distribution under unity cost ratio condition as shown below: R=
C(−|+) p(−) = = 1. p(+) C(+|−) p(+) = p(−).
To train a model with the balanced distribution from any given class distribution, simply set the training parameter M = R = p(−)/p(+) for training.
Model Complexity and Algorithm Selection in Classification Melanie Hilario University of Geneva, CSD CH-1211 Geneva 4, Switzerland [email protected]
Abstract. Building an effective classifer involves choosing the model class with the appropriate learning bias as well as the right level of complexity within that class. These two aspects have rarely been addressed together: typically, model class (or algorithm) selection is performed on the basis of default settings, while model instance (or complexity) selection is investigated within the confines of a single model class. We study the impact of model complexity on algorithm selection and show how the relative performance of candidate algorithms changes drastically with the choice of complexity parameters.
1
Introduction
The choice of the appropriate model for a given classification task comprises two complementary aspects: model class selection and model instance selection. Model class selection is usually implicit in algorithm selection, while model instance selection assumes a given model class and involves choosing the right level of complexity within that class. Though these two subproblems are intricately related, researchers have tended to focus on one to the detriment of the other. In statistical learning, emphasis is laid on model (instance) selection, defined as creating, within a specific model class, an individual model whose complexity has been fine-tuned to the given data. This viewpoint has found its fullest expression in Vapnik’s theory. The principle of structural risk minimization, which defines a trade-off between the quality of the approximation to the data and the complexity of the approximating function, relies on the existence of a structure consisting of nested subsets of functions Sk such that S1 ⊂ S2 ⊂ . . . ⊂ SN [19]. As model complexity is typically governed by one or several “capacity control” parameters, model selection in this precise sense consists in adjusting these parameters in order to generate a model instance adapted to the data. By contrast, the machine learning community lays stress on algorithm selection. In the absence of prior knowledge about the most suitable algorithm, the idea is to cover a wide range of clearly distinct learning biases to increase the chances of pinpointing a promising model class for the learning task at hand. In order to restrict the space of possible hypotheses, it has become a common expedient to evaluate algorithms with their default parameter settings (e.g., [11], [13] S. Lange, K. Satoh, and C.H. Smith (Eds.): DS 2002, LNCS 2534, pp. 113–126, 2002. c Springer-Verlag Berlin Heidelberg 2002
114
M. Hilario
and studies conducted in the Metal project [12]). The implicit assumption seems to be that once a model class with the right bias is found, it would be easier to search this more restricted subspace for the model with the adequate complexity. While intuitively appealing, this strategy runs the risk of prematurely eliminating algorithms which may be highly effective with the proper parameters but which perform poorly with their default settings. As a consequence, one can legitimately question the validity of drawing generic conclusions about an algorithm on the basis of its default behavior. Integrating algorithm selection and model selection is a long-term research goal with many open issues. In this paper, we take a tiny step in that direction by investigating the impact of parameter adjustment on algorithm selection. After a preliminary discussion of the methodology adopted to assess and compare generalization performance (section 2), we describe the experimental setup (section 3) and main observations (section 4). Finally we discuss underlying issues as well as related and future work (sections 5 and 6).
2
Comparing Performance: Weighting vs. Ranking
Predictive accuracy (or error) is perhaps the most popular measure of generalization performance, due mainly to the convenience of having a single metric as a basis for comparison. While it is satisfactory when selecting a classifier for a particular task (and assuming that time and other practical considerations are irrelevant), the accuracy criterion is fraught with problems when comparing across different task domains and levels of difficulty. Clearly, 99% accuracy in a 2-class domain with a 95%-majority class cannot be deemed better than 80% accuracy in a task involving ten equiprobable classes. In addition, selecting a single algorithm to recommend is a delicate undertaking except in fairly rare problems where one clearly dominates all the others. For these reasons, model ranking has been proposed as an alternative to categorical model selection. First, replacing absolute measures of accuracy with performance ranks facilitates transdomain comparisons; another advantage is the greater flexibility offered by a ranked set of promising alternatives rather than a single ”winner”. How are ranks established? To be at all valid, any ranking strategy should be based on statistically significant differences in the performance measure. We use the McNemar test with the Yates continuity correction to determine if one algorithm’s performance is significantly better than another’s [5]. Multiple comparisons are handled with the Bonferroni adjustment [8], but there is an additional problem raised by the transition from multiple pairwise comparisons to a common overall ranking. Simply taking the “transitive closure” of the pairwise ”better-than” judgments can lead to inconsistencies that have been addressed in Saaty’s [18] analytical hierarchy process (AHP). AHP is a comprehensive methodology for multicriteria decision-making. It is composed of three stages: hierarchical structuring of goals and criteria, pairwise comparisons, and synthesis of priorities. Our focus here is the transition from the second to the third phase, in particular the establishment of an overall order of preference based
Model Complexity and Algorithm Selection in Classification
115
on pairwise significance tests on the performance measures of candidate algorithms. The underlying scale is a set of positive numbers (typically 1-9) used to quantify judgments concerning a pair of entities. Results of pairwise comparisons of N entities are gathered in an N × N matrix A, where A(i, j) contains a numerical value expressing a relative assessment of i with respect to j, and A(j, i) contains the reciprocal of this value to indicate the inverse assessment. In other words, A(i, j) can be viewed as a ratio wi /wj of the weights given to entities i and j. In the specific task of comparing classifier performance, we use A(i, j) = A(j, i) = 1 if there is no statistically significant difference between algorithms i and j, A(i, j) = 2 if i is significantly better than j, and A(i, j) = 0.5 if j is significantly worse than i. If we knew the precise weight or importance of each of the N items, it would be straightforward to build the comparison matrix (understandably called the positive reciprocal matrix), which would then verify the equality Aw = N w. But the situation is slightly different: first, to accomodate for possible inconsistencies, we replace the constant N by an unknown λ; second, we face the inverse task of recovering the weights from the weight ratios stored in the comparison matrix. The problem is that of solving the equation (A − λI)w = 0 for w; this is no other than the classical eigenvalue problem, where w is the dominant eigenvector of the matrix(and λ = N in case of perfect consistency).We can normalize w by requiring wi = 1 with no risk of altering the weight ratios. The resulting weights give us precisely what we are looking for: a vector of priorities on the N candidate algorithms. It would be trivial to map this vector of weights onto a set of ranks. However, we would lose the relative amplitude of the weights, which is exactly the information needed to solve a problem raised by simple ranking: how can we quantify an improvement or a degradation in a classifier’s ranking under varying experimental conditions? The question is more complex than it seems, as ranks are determined by statistically significant differences in performance measures and the number of ranks varies from one situation to another. The absolute value of a classifier’s rank is meaningless in itself; an algorithm that is ranked second in a 2-rank scale is certainly worse off than one which is second in a 9-rank scale. To overcome this difficulty we can use the relative rank defined by rank(x)/#ranks. But annoying conundrums subsist, e.g., a classifier ranked first in a 2-rank scale would have the same relative rank as one that is fifth in a 10-rank scale. All of these problems are solved naturally by the use of normalized AHP weights which are by construction ranks measured on a ratio scale. We therefore propose weighting as an alternative to ranking in algorithm and model evaluation. AHP-based weights have all the known advantages of ranks plus (1) information about distances and relative magnitudes of preferences or priorities, and (2) a fixed interval of [0..1]: knowing the number of candidate algorithms, one can easily draw conclusions from an algorithm’s weight; in ranking, one needs to know in addition the effective number of significant ranks.
116
3
M. Hilario
Experimental Setup
We selected 9 learning algorithms with clearly distinct learning biases: Quinlan’s C5.0 decision tree (C5T)1 and rule inducer (C5R) [17], a multilayer perceptron (MLP) and a radial basis function network (RBF) from Clementine 5.0.1 [3], J. Gama’s linear discriminant (LDS) and Linear Tree (LTR) [7], an instance-based learner (IB1) and Naive Bayes (NB) from the MLC++ library [9], and Ripper (RIP) [4]. All experiments were done on a set of 70 selected datasets, most of them from the UCI benchmark repository. The same learning algorithms and datasets were used throughout the three experimental phases described below. 3.1
Baseline Construction
The 9 learning algorithms were run with their default parameter settings on the 70 datasets and their error rates estimated using stratified ten-fold crossvalidation. Predictions and error rates of the 9 learned classifiers were then input to the weighting procedure, which is summarized as follows. For each dataset: 1. Perform 92 pairwise comparisons using McNemar tests to determine if each learning algorithm performed significantly better (or worse) than each of the 8 others. Apply the Bonferroni adjustment and set the effective significance level to 0.05/ 92 = 0.001 to obtain a nominal significance level of 0.05. 2. Build a comparison matrix A from significance test results, with A(i, j) = 1 if there is no statistically significant difference between classifiers i and j, 2 if i is significantly better than j, and 1/2 if j is significantly worse than i. 3. Compute the eigenvalues and eigenvectors of the comparison matrix; the normalized eigenvector corresponding to the highest eigenvalue represents the weights of the 9 candidate classifiers as explained in Section 2. 3.2
Study of Parameter Adjustment
Of the 9 learning algorithms studied, 6 have one or several parameters that control the complexity of the final model. For this study we selected the principal complexity parameter of these algorithms: the degree of pruning severity for decision trees (C5T and LTR) and rules (C5R and RIP), the number of hidden units in neural networks (MLP and RBF). These 6 parameterizable algorithms will be called the target algorithms in the rest of this paper. The learning algorithms which have no complexity parameters are NB, LDS, and IB1 (an instance-based learner based on a single nearest neighbor). In C5T, C5R, and LTR, the pruning severity is governed by the c parameter which takes values in (1..99), with a default of 10 for LTR and 25 for C5T and C5R. Increasing this value decreases pruning and yields a larger tree while decreasing it results in more severe pruning. We chose to sample landmark values 1
These abbreviations will be used throughout the paper to designate the specific implementations used and not the generic algorithms.
Model Complexity and Algorithm Selection in Classification
117
10, 20, 25, 30, 40, 50, 60, 70, 80, and 90. Ripper’s S parameter controls pruning severity in the opposition direction: increasing the default value of 0.5 entails more severe pruning while decreasing it reduces pruning, yielding more complex rules. In the absence of precise documentation concerning the limits of the S parameter, we tested the following values: 0.25, 0.5, 0.75, 1, 1.25, 1.5, 1.75, 2, 10. By default, Clementine builds an RBF network with 20 hidden units; we selected the following values to compare with this default: 1, 5, 10, 30, 40, 50, 100, 200, 300. For MLP, Clementine’s default (“quick”) strategy generates a variable number of hidden units according to some internal heuristic that is not explicit in the documentation. We explored the following settings: 2, 5, 10, 20, 30, 40, 50, 100, and 200 (2 is the minimal value accepted by Clementine, and 300 was excluded since MLPs are known to require fewer hidden units than RBFNs). Altogether, 4820 ten-fold cross-validation experiments were conducted: 630 in establishing the baseline weights of 9 algorithms on 70 datasets, plus 4190 in studying the selected parameter values of the six target algorithms. 3.3
Comparison of Default and Optimal Weighted Ranks
For each dataset, each target algorithm thus generated a set of classifiers of varying complexity. The optimal model was selected according to criteria that will be specified in Section 4.2. To compare the default and optimal performance of the 6 target algorithms, we gathered the optimal models generated for each dataset together with the baseline models from the non parameterizable learning algorithms. All 9 classifiers were then reweighted using the same weighting procedure as in the baseline construction phase (Section 3.1). The main difference is that, considering the number of parameterizations explored for the 6 target 2 algorithms 62 (9 for MLP and 10 for the 5 others), we used a significance level of 0.05/ 2 = 0.000026 to obtain a nominal significance level of 0.05.
4
Results and Discussion
4.1
Baseline Observations
Table 1 gives a sampling of the weights of the 9 default classifiers on a few selected datasets. The sample datasets are ordered according to the number of distinct weights reflecting the number of significant differences detected in performance levels. The heart dataset in the first row is representative of a set of 13 “equivalent-performance” (henceforth equiperformance) datasets, so called because no algorithm significantly outperformed any other on these datasets. Of the 57 remaining datasets where error rates showed greater variability, 5 datasets produced 2 performance ranks, 31 datasets between 3 and 6, and 14 datasets between 7 and 9. Averaging over the 70 datasets, we obtain an overall weight for 2
The numbers of hidden units generated by default in MLP did not represent a distinct setting but coincided or were binned with the 9 nondefault values.
118
M. Hilario Table 1. Weights of classification algorithms with default parameter settings
Dataset
C5R
C5T
MLP
heart 0.111 0.111 0.111 crx 0.125 0.125 0.125 sonar 0.110 0.110 0.110 dermatology 0.117 0.117 0.058 glass2 0.135 0.135 0.067 clean2 0.151 0.151 0.067 processed.cleveland 2 0.102 0.095 0.110 bands 0.127 0.117 0.117 char 0.162 0.162 0.096 pyrimidines 0.180 0.154 0.103 satimage 0.103 0.095 0.142 segmentation 0.142 0.142 0.122 triazines 0.191 0.164 0.056 optdigits 0.097 0.076 0.065 waveform40 0.082 0.066 0.139
RBF
LDS
LTR
IB1
0.111 0.125 0.110 0.117 0.117 0.078 0.110 0.102 0.077 0.060 0.095 0.067 0.065 0.123 0.163
0.111 0.125 0.110 0.141 0.080 0.097 0.142 0.069 0.066 0.078 0.067 0.079 0.089 0.153 0.178
0.111 0.125 0.133 0.117 0.117 0.151 0.110 0.095 0.162 0.103 0.153 0.155 0.140 0.162 0.131
0.111 0.063 0.103 0.117 0.135 0.097 0.102 0.139 0.123 0.151 0.195 0.107 0.110 0.179 0.057
NB
RIP
0.111 0.111 0.063 0.125 0.103 0.103 0.109 0.109 0.080 0.135 0.057 0.151 0.120 0.110 0.117 0.117 0.056 0.096 0.060 0.111 0.057 0.095 0.057 0.130 0.076 0.110 0.069 0.075 0.096 0.088
#Rks 1 2 3 4 4 5 5 6 6 7 7 8 8 9 9
Summary over 70 datasets: Mean Weight Overall Rank
0.1263 0.1224 0.1122 0.0999 0.1005 0.1232 0.111 0.0917 0.1129 1 3 5 8 7 2 6 9 4
each algorithm which we can translate into integer ranks for convenience (bottom of table). However, this incurs a loss of information: relying on simple ranks, we see instantly that decision trees and rules (C5R, LTR, C5T, RIP) occupy the top four positions of the performance scale. Weights reveal more subtle differences: for instance, the distance between C5R and LTR is four times that between LTR and C5T while the distance between C5T and RIP is 13 times greater than that between LTR and C5T.
Fig. 1. Mean weights over 70 datasets of learning algorithms with default parameters
Figure 1 visualizes relative weighting leaps which are completely hidden by the notion of ranks. It is however a bit awkward to express relative performance in terms of pairwise differences between algorithm weights. Since the weights of
Model Complexity and Algorithm Selection in Classification
119
N candidate algorithms sum to unity, a convenient reference is the equiperformance level, i.e.. 1/N (0.111 for our 9 algorithms, just a hair’s breadth above the 0.110 line in Fig. 1). The equiperformance line plays the same role as the wellknown default accuracy in comparing generalization performance, but it has the clear advantage of being data-independent and therefore usable in cross-domain comparisons. While all 4 decision tree and rule learners are above the equiperformance line in our example, C5R is about 8 times farther from it than RIP. Note that from the viewpoint of both ranks and weights, the neural networks are clearly outperformed by decision trees and rules; MLP and RBF rank 5th and 8th respectively; MLP is barely above the equiperformance line whereas RBF is clearly below it, along with LDS and NB. 4.2
Impact of Parameter Variation
We ran each target algorithm with 10 different parameter settings on each dataset. We then selected the optimal parameter, defined as that which led to the lowest 10-fold cross-validation error3 —and, in the rather frequent case of ties, to the smallest model4 . This section focuses mainly on how each algorithm’s performance varies with the complexity parameter; inter-algorithm conclusions will be discussed in the next section. Figure 2 shows the distribution of the optimal parameter for the four decision tree/rule learners. The height of the bar representing a particular parameter setting indicates the number of datasets for which that setting produced the lowest error. In each chart, the figures depicted by the bars sum up to several times the number of datasets due to numerous ties in optimal performance. The darker part of each bar shows the number of datasets for which a given parameter value minimizes both error and complexity. The charts of the recursive partitioning algorithms (C5.0 trees and rules, Ltree) show a clear pattern: simpler models achieve higher generalization accuracy. In C5.0 trees, for instance, a c value of 10 (most severe pruning) sufficed to attain the minimal error in 36 of the 70 datasets. The default parameter of c=25 led to the lowest error on 23 of the 70 datasets, but in 17 of these 23 cases the same error was produced by a smaller c value (which actually produced the same model). On 28 datasets, lower c values (and smaller trees) produced a lower error than the default (e.g., waveform21, waveform40, dna-splice, clean2, and satimage). In these cases, the default parameter was clearly overfitting the data. By contrast, the default did not yield the complexity needed to classify 18 datasets, on which maximal accuracy was attained only with c values ranging from 30 to 70 (e.g., bands, char, sonar, triazines). The behavior of the complexity parameter in C5.0-rules is rather similar, except that the distribution is a bit less sparse at the higher end of the c pa3 4
At this stage use of the error rate is reasonable since different models are being compared on the same dataset. The model size depends on the representation used and can be used only in comparing different models produced by the same algorithm (e.g., # of leaf nodes in decision trees, # of hidden nodes in neural networks).
120
M. Hilario
Fig. 2. Distribution of optimal complexity parameters of C5.0-tree, C5.0-rules, Ltree, and Ripper (clockwise)
rameter spectrum. In particular, 5 datasets required a c value of 80 and above to attain optimal performance in C5.0-rules, as opposed to none in C5.0 trees (however the error thus obtained was smaller than that of the C5.0-tree optimal model). The distribution of parameter optima in Ltree is strikingly similar to that of C5.0, with one major difference: the default c value in Ltree is 10, which actually attained the highest accuracy on the majority of the datasets. Finally, Ripper’s behavior is clearly atypical; contrary to right-skewness of the three other distributions, optimality appears to be more uniformly distributed among the different S values; note however the two rather distant modes, S=1.75 and S=0.5, the default setting which proves optimal in 18 datasets. Figure 3 shows the distribution of the optimal complexity parameters which correspond roughly to the number of hidden units in Clementine RBF and MLP. In RBF, the default number of hidden units (20) proved optimal in only 13 datasets. In 15 datasets, RBF obtained equal or better accuracy with much simpler networks comprising 1 or 10 hidden units. In 34 datasets, it took a minimum of 100 units to achieve optimal performance, the overall mode being situated at c=200 units. For multilayer perceptrons, we saw in Section 3.2 that Clementine follows a more flexible (albeit more opaque) default strategy in determining the number of hidden units. On the 70 experimental datasets, Clementine’s quick strategy
Model Complexity and Algorithm Selection in Classification
121
Fig. 3. Distribution of optimal number of hidden units in Clementine RBFN (left) and MLP (right)
produced default values ranging from a minimum of 4 (for simple problems such as iris, breast-cancer-wisconsin, and the monks problems) to upper limits of 61 units for triazine, 66 for dna-splice, and 67 for clean2. The default value turned out to be optimal in only 9 of the 70 datasets. In 16 cases, Clementine selected h values of 5 and 10 whereas 2 hidden units sufficed to achieve equivalent or even significantly better performance. Altogether, the optimal values were smaller than the default values in 28 cases and larger in 33 cases. It is also instructive to compare the distributions of the optimized number of hidden units of MLP and RBF. Figure 3 shows a right skewed distribution of optimal values (minimal error at minimal size) for MLP, with highest frequencies between 2 and 10 hidden units. RBF displays a multimodal distribution with modes at or near the extremes. However, values at the higher end (200-300) clearly outnumber those at the lower end (1-10). On approximate half of the datasets the most accurate RBF models have a topology of 100 hidden units or more. This confirms the well-known fact that RBF networks require more hidden units than multilayer perceptrons. 4.3
The Final Picture
After selecting the optimal models produced by the parameterizable algorithms for each dataset, we reapplied the McNemar-Bonferroni-AHP weighting procedure to these classifiers together with those generated by the 3 non parameterizable algorithms. To give an idea of the changes in performance entailed by parameter selection, Table 2 shows the new weights obtained on the sample datasets of Table 1. The overall picture is summarized at the bottom of the table, which shows the mean weight (and simplified rank) of each algorithm over 70 datasets. These figures, compared with those in Table 1, reveal a dramatic change in the performance of the neural networks with respect to the baseline. Quickly said, MLP jumps from fifth to the top rank and RBF from rank 8 to rank 5. Expressed more
122
M. Hilario Table 2. Weights of classification algorithms after parameter optimization
Dataset
C5R
C5T
MLP
RBF
LDS
heart 0.111 0.111 0.111 0.111 0.111 crx 0.125 0.125 0.125 0.125 0.125 sonar 0.111 0.111 0.111 0.122 0.111 dermatology 0.109 0.109 0.129 0.129 0.129 glass2 0.127 0.127 0.081 0.109 0.102 clean2 0.132 0.132 0.158 0.115 0.072 processed.cleveland 2 0.111 0.111 0.111 0.111 0.111 bands 0.119 0.110 0.119 0.119 0.084 char 0.142 0.154 0.132 0.105 0.067 pyrimidines 0.197 0.120 0.120 0.072 0.072 satimage 0.096 0.096 0.164 0.164 0.067 segmentation 0.131 0.144 0.131 0.067 0.079 triazines 0.191 0.149 0.089 0.065 0.078 optdigits 0.076 0.065 0.166 0.195 0.111 waveform40 0.088 0.067 0.163 0.163 0.163
LTR
IB1
0.111 0.110 0.111 0.109 0.109 0.132 0.111 0.110 0.154 0.120 0.096 0.144 0.149 0.130 0.123
0.111 0.069 0.111 0.109 0.127 0.079 0.111 0.119 0.098 0.120 0.164 0.115 0.104 0.118 0.057
NB
RIP
0.111 0.111 0.069 0.125 0.104 0.111 0.088 0.088 0.081 0.138 0.058 0.123 0.111 0.111 0.110 0.110 0.057 0.090 0.058 0.120 0.058 0.096 0.058 0.131 0.056 0.121 0.069 0.069 0.088 0.088
#Rks 1 3 3 3 4 7 1 3 8 4 4 6 8 8 5
Summary over 70 datasets: Mean Weight Overall Rank
0.1200 0.1173 0.1211 0.1138 0.0984 0.1205 0.1048 0.0920 0.1121 3 4 1 5 8 2 7 9 6
precisely in terms of performance weights, MLP’s distance from the equiperformance level increases by an order of magnitude (from 0.001 to 0.01) while that of RBF shifts from -0.01 to +0.002. Despite its effective default parameter value, LTR still improves with complexity adjustment, outperforming both variants of C5.0. The two rule learners (C5R and RIP) find themselves less favorably weighted despite improved performance, but this is due mainly to the relatively more significant improvement of MLP, RBF, and LTR. Finally, the three nonparameterizable algorithms end up at the bottom of the performance scale. Note, however, that IB1 is much closer to the default performance than LDS and NB (Fig. 4, right); a possible explanation is that IB1, as a high-variance algorithm (as opposed to the other two—high-bias—learners), can adapt more easily to complex problems.
Fig. 4. Mean weights of algorithms with default (left) and optimized (right) parameters
Model Complexity and Algorithm Selection in Classification
123
An intriguing result of the final weighting process is the increase in the number of equiperformance cases: the optimized models display no significant difference in performance on 23 datasets (against 13 in the baseline evaluation). This is illustrated in the table by processed.cleveland 2, on which the 5 distinct initial weights flattened out to a single common weight. The trend towards a decrease in significant differences is confirmed in the general case, as can be seen by comparing the rightmost columns of Tables 1 and 2 (boldface). More clearly, the final performance weights of the optimized algorithms no longer display the abrupt leaps observed in the default settings (Fig. 4). There are two possible explanations for this: the convergence of the optimized models’ accuracies in the vicinity of the Bayes error, and/or the extremely stringent significance level of 0.000026 imposed by the Bonferroni adjustment.
5
Discussion and Related Work
The weighting/ranking upheaval entailed by parameter selection demonstrates the inconclusive nature of comparative evaluations based on default settings. By blending model instance selection into algorithm selection, we can more confidently draw conclusions about the specific biases and areas of expertise of the different algorithms. We tried the meta-learning approach to uncover a few clues on why a properly parameterized algorithm outperforms the others on given dataset. The task was cast as one of binary classification: a dataset is considered to belong to a learning algorithm’s area of competence if the algorithm obtained the highest weight (possibly tied with others) on the dataset. The input meta-data consisted of 20 dataset characteristics (e.g., # of instances, classes, and explanatory variables; proportion of missing values, average entropy of predictive variables, class entropy, mutual information between predictive and class variables). Since the objective was gain of insight rather than predictive accuracy, we chose C5.0-rules as our meta-learner. Examples of learned rules are given in the table below. The head designates the algorithm(s) whose success can be explained by the corresponding conditions. Each rule is followed by its coverage and confidence (correctly covered cases). Head MLP LTR C5R C5R RP NN NN UP
Conditions P ≤ 20 & mean attribute entropy ≤ 5.24 P/K ≤ 16 & default accuracy > 0.53 default accuracy > 0.16 & mean attribute entropy ≤ 4.48 P ≤ 32 & default accuracy > 0.16 & equiv#atts ≤ 1.71 P > 8 & P/K ≤ 16 & default accuracy > 0.53 default accuracy > 0.68 & mean NMI ≤0.01 & dsize ≤ 43890 P ≤ 40 and default accuracy > 0.16 and mean NMI > 0.01 N ≤ 339 & P ≤ 40
Cover. 34 37 49 5 31 11 22 21
Conf. 0.944 0.846 0.824 0.857 0.879 0.923 0.875 0.783
In the above table, N is the number of instances, K the number of classes, and P the number of predictive attributes. The first four rules explain the performance of the three top-ranking algorithms after parameter optimization. The
124
M. Hilario
most commonly used meta-features are those related to the difficulty of the problem (default accuracy), the dataset size (P,P/K or the attribute/class ratio and dsize = P × N ) and information content (mean attribute entropy and equiv#atts or the ratio of the class entropy to the mean mutual information between class and predictive attributes). Another challenge was to explain the striking correlation in the performance of the three recursive partitioning algorithms: C5R, C5T and LTR obtained the same (highest) weights on 38 out of the 70 datasets. The RP rule shown in the table correctly explains 28 of these using the same meta-features previously described. To explain a similar phenomenon, the equality of weights of MLP and RBF on 39 datasets, the NN rules use an additional characteristic, the mean NMI or average mutual information between attributes and the class variable, normalized by the average number of distinct values per attribute. Finally, it was tempting to find an explanation for equivalent performance, whose practical consequence is the expendability of algorithm or model selection for datasets that satisfy the induced conditions. The UP rule in the table expresses the idea that below a certain dataset size no algorithm can demonstrate its superiority over the other. Datasets correctly covered by this rule include staples of the UCI repository such as balance-scale, iris, heart, hepatitits, and the different versions of the Wisconsin breast-cancer database. While this rule does make sense, an alternative explanation could be, as some have conjectured, that many existing algorithms have been overfitted to these classical UCI benchmarks. The choice of the appropriate model complexity has been a long-standing subject of research in the neural network community. Aside from many futile attempts to discover simple heuristics based on, say, the number of inputs and outputs, researchers have explored two main avenues: (1) strategies for iteratively adjusting model complexity by varying the number of hidden units/layers and/or a regularization term, typically via a double cross-validation loop [2][16]; and (2) so-called self-configuring networks, where the number of hidden units (and layers in certain cases) is dynamically incremented (e.g., [6][1]) or pruned (e.g., [10] [14]) during training. However, there has been relatively little work on the impact of complexity parameters in algorithm selection. As mentioned in Section 1, most large-scale comparative studies use default parameters; while performance is determined as much by these parameters as by the algorithms’ inherent biases, observed results have usually been attributed to the latter. In a few cases, algorithm comparisons are slightly broadened to take account of parameter effects. For instance, in Lim et al.’s [11] study of 33 classifiers, roughly a third are actually alternative parameterizations of a single learning algorithm; however, these variants were viewed as distinct learning algorithms in the comparative analysis.
6
Limitations and Future Work
In this paper we investigated the impact of model (instance) selection on algorithm selection. We ran 9 learning algorithms with their default parameters on
Model Complexity and Algorithm Selection in Classification
125
70 datasets, then experimented with 9-10 different settings on the six algorithms which could induce models of varying complexity. To track performance variations more effectively, we proposed weighting as an alternative to ranking in the comparative study of algorithms and models. We showed that a number of problems raised by the use of ranks are smoothly solved by AHP-based weights. In analyzing observed results, we used rule meta-learning as an auxiliary tool in explaining certain observations on the basis of dataset and task characteristics This study is an initial foray into the problem of the interaction between model complexity and algorithm selection, and is of necessity still crude and partial. One obvious limitation is that it focuses on a single complexity parameter for each of the algorithms studied, ignoring essential interactions between different model and runtime parameters (e.g., between the number of hidden units and the width of each unit’s region of influence in RBF, or between network topology and the number of training cycles in MLPs). Our experiments have shown clearly that we cannot rely on comparative evaluations based on default parameter settings. At the same time, in many real-world applications we cannot afford the luxury of time-consuming experimentation on each candidate algorithm. Another practical research objective is thus the development of more adequate default strategies for handling parameters data mining tools. While such automated strategies will never replace empirical fine-tuning to the application data, they will at least provide a more adequate baseline to start from, thus accelerating the modeling process. From the results of this study, neural network algorithms stand to gain most from the development of such strategies. Finally, this study uses Saaty’s technique for deriving weights from pairwise comparison matrices, but his analytical hierarchy process has a much broader application scope in decision theory and its full potential has yet to be exploited in data mining. In particular, AHP offers a principled way of synthesizing weighted priorities in the presence of multiple—possibly conflicting—goals and criteria (e.g., generalization performance, computational cost, interpretability, and practicality). It would be certainly be interesting to compare the hierarchical approach with data envelopment analysis, another technique from operational research which has been adapted to the integration of multiple criteria in knowledge discovery [15].
References 1. E. Alpaydin. GAL : Networks that grow when they learn and shrink when they forget. Technical Report TR-91-032, International Computer Science Institute, Berkeley, May 1991. 2. C. M. Bishop. Neural Networks for Pattern Recognition. Oxford University Press, 1995. 3. Clementine. http://www.spss.com. 4. W. W. Cohen. Fast effective rule induction. In Proc. 11th International Conference on Machine Learning, pages 115–123, 1995.
126
M. Hilario
5. T. G. Dietterich. Statistical tests for comparaing supervised classification learning algorithms. Technical report, DCS, Oregon State University, 1996. 6. S. E. Fahlman and C. Lebiere. The Cascade-Correlation learning architecture. Technical Report CMU-CS-90-100, Carnegie Mellon University, 1990. 7. J. Gama and P. Brazdil. Linear tree. Intelligent Data Analysis, 3:1–22, 1999. 8. D. D. Jensen and P. R. Cohen. Multiple comparisons in induction algorithms. Machine Learning, 38(309–338), 2000. 9. R. Kohavi, G. John, R. Long, D. Manley, and K. Pfleger. MLC++: A machine learning library in C++. Technical report, CSD, Stanford University, 1994. 10. Yann LeCun, J. S. Denker, and S. A. Solla. Optimal brain damage. In Advances in Neural Information Processing, 2, pages 598–605, 1990. 11. T. Lim, W. Loh, and Y. Shih. A comparison of prediction accuracy, complexity, and training time of thirty-three old and new classification algorithms. Machine Learning, 40:35–75, 2000. 12. MetaL Consortium. Project Homepage. http://www.metal-kdd.org/. 13. D. Michie, D. J. Spiegelhalter, and C. C. Taylor, editors. Machine learning, neural and statistical classification. Ellis-Horwood, 1994. 14. M. C. Mozer and P. Smolensky. Skeletonization: a technique for trimming the fat from a network via relevance assessment. In D. S. Touretzky, editor, Advances in Neural Information Processing, 1, pages 107–116, San Mateo, CA, 1989. Morgan Kaufmann. 15. G. Nakhaeizadeh and A. Schnabl. Development of multi-criteria metrics for evaluation of data mining algorithms. In Proc. Third Intl. Conf. on Knowledge Discovery and Data Mining, pages 37–42, 1997. 16. B. D. Ripley. Statistical ideas for selecting network architectures. In B. Kappen and S. Gielen, editors, Neural Networks: Artificial Intelligence and Industrial Applications, pages 183–190, London, 1995. Springer. 17. http://www.rulequest.com. 18. T. L. Saaty. Fundamentals of Decision Making and Priority Theory. RWS Publications, 1994. 19. V. Vapnik. The Nature of Statistical Learning Theory. Springer, 1995.
Experiments with Projection Learning Tapio Elomaa and J.T. Lindgren Department of Computer Science P. O. Box 26 (Teollisuuskatu 23) FIN-00014 University of Helsinki, Finland {elomaa,jtlindgr}@cs.helsinki.fi
Abstract. Excessive information is known to degrade the classification performance of many machine learning algorithms. Attribute-efficient learning algorithms can tolerate irrelevant attributes without their performance being affected too much. Valiant’s projection learning is a way to combine such algorithms so that this desired property is maintained. The archetype attribute-efficient learning algorithm Winnow and, especially, combinations of Winnow have turned out empirically successful in domains containing many attributes. However, projection learning as proposed by Valiant has not yet been evaluated empirically. We study how projection learning relates to using Winnow as such and with an extended set of attributes. We also compare projection learning with decision tree learning and Na¨ive Bayes on UCI data sets. Projection learning systematically enhances the classification accuracy of Winnow, but the cost in time and space consumption can be high. Balanced Winnow seems to be a better alternative than the basic algorithm for learning the projection hypotheses. However, Balanced Winnow is not well suited for learning the second level (projective disjunction) hypothesis. The on-line approach projection learning does not fall far behind in classification accuracy from batch algorithms such as decision tree learning and Na¨ive Bayes on the UCI data sets that we used.
1
Introduction
Redundancy and excess of information are typical in natural information processing. Unfortunately, from the algorithmic point of view, such expendable information is often directly reflected in the efficiency of data processing. Moreover, the quality of the hypothesis is known to degrade for many learning algorithms through the addition of irrelevant data. One approach to tolerating excessive information in algorithmic learning is attribute-efficient learning [6,10,11]. This approach allows a learning algorithm to make a polynomial number of prediction errors in the number of relevant features, but only a subpolynomial number in the total number of features, before converging to the correct concept. The setting for attribute-efficient learning is on-line learning, where the (training) instances are received one at a time, the current hypothesis is used to predict the class of an instance before receiving a reinforcement (the true class of the instance), after which the hypothesis is updated on the basis of the S. Lange, K. Satoh, and C.H. Smith (Eds.): DS 2002, LNCS 2534, pp. 127–140, 2002. c Springer-Verlag Berlin Heidelberg 2002
128
T. Elomaa and J.T. Lindgren
difference, if any, in the prediction and the reinforcement. In on-line learning a natural goal is to attempt to minimize the number of classification errors of the learner. This learning model is known as the mistake bound model [10]. Attribute-efficiency is particularly important in domains where there are huge numbers of attributes, out of which only a few are relevant [9]. Examples of such domains are text classification [5], natural language processing [8], and object recognition [14]. Attribute-efficiency can typically be guaranteed only for very restricted concept classes. The best-known attribute-efficient learning algorithm is Littlestone’s [10] Winnow — a variant of the classical Perceptron algorithm [13]. Also other attribute-efficient learning algorithms are very simple algorithms that alone have a limited computational power [18]. Combining linear machines into a neural network [7] or using linear classifiers in the implicit feature space of support vector machines [22] makes these simple and restricted learning machines a powerful method to use. How to combine linear classifiers so that their attribute-efficient properties are maintained is the question that projection learning [20] wants to tackle. Even though variants of Winnow and networks of Winnows — some even using projections to guide the examples to subclassifiers — have proved successful in practical learning tasks [2,5,8,14], the two-level projection learning proposed by Valiant [20] has not been tested empirically. Our aim is to study, on one hand, how much advantage does projection learning bring in contrast to using Winnow alone and, on the other hand, how close to such empirically successful algorithms as C4.5 decision tree learner [12] and Na¨ive Bayes [3] does projection learning reach. Our study is experimental, we test the different learning algorithms using UCI data sets [1]. Out of the projection learning versions that we have tested, the most successful is the one in which the lower level learners are Balanced Winnows [11] and the high-level learner is the standard (positive) Winnow. As projection sets both single-attribute projections and quadratic ones are useful. Statistically significant improvement over using Winnow alone is recorded on many UCI domains. However, the loss in time- and space-efficiency can be high. The classification accuracy of the on-line method approaches that of decision tree learning and Na¨ive Bayes. The remainder of this article is organized as follows. In Section 2 we recapitulate the basics of attribute-efficient learning. In particular, the Winnow algorithm is examined. Section 3 concentrates on projection learning. We present our empirical experiments in Section 4. The last section gives the concluding remarks of this study.
2
Mistake-Bounded Learning and Attribute-Efficiency
Let us consider Boolean functions on n variables x1 , . . . , xn . A concept is a n mapping from value assignments (instances) Xn = { 0, 1 } to { 0, 1 } for some finite n. Those instances that are mapped to 1 are called positive and those mapped to 0 negative. A concept class C is the union of stratified subclasses Cn ,
Experiments with Projection Learning
129
n≥1 Cn , where c ∈ Cn has domain Xn . An instance x ∈ Xn together with its assigned class y = c(x) makes up an example (x, y). A collection S of (possibly multiply occurring) examples is called a sample. The size of a sample |S| is the total number of examples in it. The underlying concept c determining the classification of instances is the target concept that we try to learn. By ES (L) we denote the number of errors that learning algorithm L makes on sample S with respect to the target concept c.
2.1
Online Learning and Attribute-Efficient Learning
In on-line learning one wants to update the hypothesis incrementally and efficiently at the same time minimizing the number of erroneous predictions for instances. Many on-line algorithms — like Perceptron — are conservative in the sense that the hypothesis is not updated when the label of the instance was predicted right. Only falsely classified instances cause the algorithm to make changes to its guess about the correct concept. Assuming that the on-line learning algorithm will eventually converge to the correct hypothesis, we can set a mistake bound for the learning algorithm [10]. Now, learning within the mistake bound model is defined through the number of errors made before converging to the correct concept. Definition 1. Learning algorithm L learns the concept c within the mistake bound model, if for all samples S that are consistent with c it holds that ES (L) ≤ poly(n). Given examples described by n attributes, it is often the case that only few of the attributes are relevant in determining the value of the class attribute. Those attributes that have no effect are called irrelevant attributes. If the efficiency of our learning algorithm depends linearly (or through a higher order dependency) on the number of attributes used to describe the instances, utmost care must be exercised in choosing the variables to represent the data. A more natural approach is to have the learning algorithm efficiently recognize the relevant attributes from among the multitude of possibilities. Definition 2. Learning algorithm L is attribute-efficient in the mistake bound model, if ES (L) ≤ poly(k)polylog(n), where n is the total number of attributes and k is the number of relevant attributes. Winnow [10] (see Table 1) is the best-known attribute efficient learning algorithm. Like Perceptron it processes the given sample S one example (x, y) at a time updating the weight vector w ∈ Rn each time an example gets misclassified. The algorithm predicts the instance x to be positive if x · w > θ, where θ ∈ R is a threshold value given as an external parameter. If the inner product x · w does not exceed θ, then the prediction is h(x) = 0. After predicting the class of x its true class y is revealed, and the hyperplane hypothesis defined by the weight vector w is updated. Only falsely classified instances require changing the hypothesis. Each weight wi that corresponds to
130
T. Elomaa and J.T. Lindgren Table 1. The linear learning algorithm Winnow [10,11].
Winnow( S, α, θ ) % input: sample S of m pairs (x, y) and positive reals α and θ % maintain hypothesis: h(x) = sgn(w · x > θ) { initialize w ∈ Rn with positive values; for each example (x, y) ∈ S { predict h(x) = sgn(w · x > θ); for each xi = 1 { update wi ← wi α(y−h(x)) ; } } }
an active attribute xi = 1 in the misclassified instance is changed. A false positive prediction (h(x) = 1 and y = 0) causes active weights to be demoted and a false negative (h(x) = 0 and y = 1) causes them to be promoted. However, the updates in Winnow are multiplicative rather than additive as in Perceptron. Thus, whenever xi = 1, wi /α, when false positive is predicted; (y−h(x)) = wi ← wi α wi α, when false negative is predicted. For monotone disjunctive concepts on k variables Winnow can be shown to learn the correct concept from n-dimensional inputs with at most 2+3k(1+log n) mistakes before converging to the correct concept by setting α = 2, θ = n, and the initial value of w to be 2 [10]. Thus, for this concept class Winnow is attribute efficient. For the Perceptron algorithm the number of mistakes for monotone disjunctive concepts grows linearly in n [9]. The fact that Winnow can learn monotone disjunctive concepts attributeefficiently does not, however, mean that it would be attribute-efficient for all linear concept classes. For example, in leaning arbitrary linear threshold functions Winnow is not attribute efficient [15]. It can even make an exponential number of mistakes for some concept classes [16]. 2.2
Balanced Winnow
Winnow initializes the weight vector w with positive initial values. If some of the weights are never updated, they will have a positive bias in classification. Balanced Winnow [11] neutralizes this effect by maintaining, in addition to w, another weight vector v ∈ Rn , which is updated conversely to w. Thus, in v attributes that correlate negatively with positive examples should have high weights. The algorithm predicts positive class whenever w · x > v · x. Hence, in Balanced Winnow the weights of those attributes that have not been updated cancel each other out. This algorithm does not require the threshold parameter
Experiments with Projection Learning
131
θ, because classification is decided by the two weight vectors. Taking the negatively correlating attributes into account extends the class of attribute-efficiently learned concepts from monotone to arbitrary disjunctions.
3
Projection Learning
Linear classifiers such as the Perceptron and Winnow have serious fundamental and practical limitations in their expressive power, even if they have been successfully applied in some domains [2,5,8,14]. Valiant [20] has proposed projection learning as an approach to enrich the expressive power of attribute-efficient learners without losing their essential properties. The need for more expressive attribute-efficient learners arises in computational models of cognitive systems [19,21]. 3.1
Valiant’s Projection Learning Algorithm Y
A projection (or a restriction) ρ of Xn is a subset of Xn [20]. Usually a projection is represented by a simple constraint on the original attributes; e.g., x1 = 1 or x2 = x3 . For a function f : Xn → { 0, 1 } the restriction fρ of f is defined as f (x), when ρ(x) = 1; fρ (x) = 0, otherwise. Hence, fρ (x) = ρ(x)f (x). Let R = { ρ1 , . . . , ρr } be a set of projections. The projection set R could be for example the set of quadratic projections { xi xj | 1 ≤ i < j ≤ n }. A projective disjunction over (C, R) is a function c of the form c(x) = ρ1 (x)c1 (x) ∨ ρ2 (x)c2 (x) ∨ . . . ∨ ρm (x)cm (x), where ρ1 , . . . , ρm ∈ R and c1 , . . . , cm ∈ C and ρi c = ρi ci for each i, 1 ≤ i ≤ m. Thus, it cannot happen for any x that ρi (x) = c(x) = 1, ci (x) = 0, and cj (x) = ρj (x) = 1 for some i = j. In order to learn a projective disjunction on concept class C one can, for each ρ, apply a learning algorithm to those examples that satisfy ρ to obtain a hypothesis hρ . A higher level hypothesis h learns to distinguish the relevant projection hypotheses from those that are not useful in learning the target concept. If the concept class C can be learned attribute-efficiently, then also projective disjunctions on it can be learned efficiently. The projection learning algorithm Y [20] (see Table 2) uses learning algorithm A to learn hypotheses for the projections and (possibly) another algorithm B for learning the projective disjunction. For each example (x, y) the algorithm r composes a meta-instance z ∈ { 0, 1 } by querying what is the classification of the projection hypotheses hρ on x if x satisfies the restriction ρ; i.e., z = (ρ1 (x)hρ1 (x), ρ2 (x)hρ2 (x), . . . , ρr (x)hρr (x)).
132
T. Elomaa and J.T. Lindgren Table 2. Valiant’s [20] projection learning algorithm Y.
YA,B ( S, R ) % A and B are learning algorithms % input: sample S of m pairs (x, y) and a set of projections R % maintain hypothesis hρ for each ρ ∈ R and % h for the projective disjunction over R { for each ρ ∈ R { initializeA ( hρ ); } initializeB ( h ); for each example (x, y) ∈ S { for each ρ ∈ R { zρ ← ρ(x)hρ (x); } predict h(z); for each ρ ∈ R { if ( ρ(x) = 1 ) updateA ( hρ , (x, y) ); } for each ρ ∈ R { zρ ← ρ(x)hρ (x); } updateB ( h, (z , y) ); } }
This meta-instance is then classified using the hypothesis for the projective disjunction h. Once the prediction is obtained, projection hypotheses hρ can be updated according to the learning algorithm A that has been used to learn them. The original example (x, y) is used to update each hypothesis corresponding to a satisfied projection; a hypothesis corresponding to an unsatisfied projection does not even have to be evaluated. After the update a new meta-instance z is constructed for updating the hypothesis h corresponding to the projective disjunction. This time the update uses (z , y) and is based on the learning algorithm B, which is used for learning h. Valiant [20] has shown that the algorithm Y, using Winnow as the learning algorithms A and B, has a mistake bound O(sk log n+s log r+s) in learning projective disjunctions over a concept class that can be learned attribute-efficiently, where n and r are the numbers of all attributes and projections, respectively, and k and s are the numbers of relevant attributes and projections, respectively. The logarithmic dependency on the total number of projections makes Y projectionefficient in the same sense as Winnow is attribute-efficient. Moreover, it preserves the basic attribute-efficiency. 3.2
On the Efficiency and Expressive Power of Projection Learning
Projection learning can be quite time consuming in practice. Consider, for example, quadratic projections. There are O(n2 ) variable pairs. For each instance x we have to check, in the worst case, whether it satisfies these O(n2 ) pairs or not. Furthermore, we have to update all projection hypotheses. It is reasonable to
Experiments with Projection Learning
133
assume that the update takes O(n) time. Thus, the time of Y in using quadratic projections can be as much as O(n3 ) for each example. Also the space requirements of Y can be high. Consider, again, quadratic projections. If n = 100, there will be n(n − 1)/2 ≈ 5, 000 variable pairs. Initializing a Winnow classifier for each such projection will mean that roughly 500, 000 weights have to be maintained. On the other hand, adding the quadratic projections as new features into the input space would have meant that only approximately 5, 100 features would have been enough. Valiant’s [20,21] motivation for learning projective disjunctions is to extend the class of (practical) concepts that can be learned attribute efficiently. A full concept may be too hard to learn attribute-efficiently by Winnow alone, but in practice it may suffice that restricted parts of different concepts are known. In principle projection learning has a very high expressive power, when allowed powerful enough projection functions. One can show that Y can (trivially) learn in principle any consistent concept: Proposition 1. Assuming a suitable set of projections, the algorithm Y learns any target concept c that is consistent with the sample S, if such a concept exists. Proof. Let projections ρ1 and ρ2 be such that ρ1 (x) = 1 if and only if c(x) = 1 and ρ2 (x) = 0 if and only if c(x) = 0. Then all positive examples will be handled by hρ1 and the negative ones by hρ2 . Therefore, eventually hρ1 (x) = 1 for any x such that ρ1 (x) = 1 and hρ2 (x) = 0 for any x such that ρ2 (x) = 1. Thus, ρ1 (x)hρ1 (x) ∨ ρ2 (x)hρ2 (x) ≡ ρ1 (x) ∨ 0 ≡ ρ1 (x) ≡ c(x). Projection learning closely resembles stacked generalization [24]. Also in the latter a main learner is trained by meta-examples composed from the predictions of subclassifiers. However, in Y the projections guide the input to the subclassifiers. Nevertheless, it is natural to assume that some of the observations concerning stacked generalization might hold for projection learning as well. For example, it is known that stacking will overfit if the meta-examples used to train the main classifier are constructed from the training examples that have been used to train the subclassifiers [17].
4
Empirical Evaluation
In this section we compare Winnow, Balanced Winnow, and projection learning algorithm Y to the standard baseline methods, decision tree learning and Na¨ive Bayes (NB). As the decision tree learning algorithm we use the C4.5 clone j48 [23]. The algorithms are evaluated on well-known UCI datasets [1]. Numeric attributes were discretized using Fayyad and Irani’s [4] greedy method, even though the simple equal width binning method did not significantly degrade the accuracies of the attribute efficient algorithms. This robustness held even when the number of bins was only roughly guessed. On the other hand, careless discretization caused j48 and NB to perform significantly worse.
134
T. Elomaa and J.T. Lindgren
Table 3. Winnow (W) versus Balanced Winnow (BW), j48, Na¨ive Bayes, and Y2B . W BW j48 NB Dataset Breast cancer 66.87 68.78 73.87 ◦ 73.12 ◦ Breast Wisconsin 95.17 90.45 • 94.69 97.25 Credit rating 77.12 80.26 ◦ 86.17 ◦ 85.72 ◦ German credit 69.09 70.53 ◦ 71.54 ◦ 74.22 ◦ Heart statlog 77.19 69.74 • 81.56 ◦ 83.00 ◦ Hepatitis 76.86 79.67 ◦ 78.65 83.71 ◦ Horse colic 75.79 76.22 85.25 ◦ 79.23 ◦ Ionosphere 83.10 84.25 92.74 ◦ 90.43 ◦ Kr-vs-kp 59.67 73.62 ◦ 99.40 ◦ 87.84 ◦ Labor 79.93 85.47 ◦ 78.43 89.07 ◦ Pima diabetes 70.79 64.31 • 74.53 ◦ 75.03 ◦ Sick 88.14 95.18 ◦ 97.79 ◦ 96.37 ◦ Sonar 73.29 66.43 • 75.23 76.87 ◦ Vote 91.28 92.12 96.46 ◦ 90.19 ◦, • statistically significant improvement or degradation
Y2B 64.43 96.61 83.23 ◦ 70.09 ◦ 81.37 ◦ 75.97 79.91 ◦ 89.57 ◦ 78.06 ◦ 87.10 ◦ 72.84 ◦ 93.98 ◦ 74.75 94.89
Another important issue is choosing good learning parameters for the algorithms. To be fair, we apply the usual settings. For Winnow-based algorithms, this means the values providing attribute-efficiency when learning disjunctions, that is, learning rate α = 2, threshold θ = |x| = n and equal starting weights wi = 2 for all i ∈ { 1, . . . , n }. For j48, the decision tree is post-pruned with subtree raising using confidence threshold of 0.25. In addition, each leaf that does not represent more than one training example is pruned. Our results are based on ten-fold cross-validation with accuracies averaged over ten runs. Statistical significance was measured with the paired t-test at confidence level 0.05. 4.1
Feasibility of Projection Learning
The question that we consider first is the feasibility of projection learning in the form of algorithm Y. We are particularly interested in the case where some simple set of projections is chosen without domain knowledge. We want to know whether the on-line algorithm Y can compete in prediction accuracy with the well-known learning algorithms in this setting. Here Y2B denotes algorithm Y using the set of quadratic projections as R and algorithms Winnow and Balanced Winnow as on-line learners B and A, respectively. The results in Table 3 clearly show that even though the basic attributeefficient methods are not very accurate on these problems, the more advanced algorithm Y is able to find significantly more accurate hypotheses. It still mostly loses to the baseline algorithms j48 and NB, but attains an overall performance much closer to their level than the linear classifiers alone do.
Experiments with Projection Learning
135
Table 4. Effect of projection sets and algorithm combinations on Y. Dataset Y1B Y2 W Y1 66.87 66.38 68.67 ◦ 63.88 Breast cancer Breast Wisconsin 95.17 95.74 95.63 96.14 Credit rating 77.12 80.23 ◦ 82.46 ◦ 80.46 ◦ German credit 69.09 68.67 70.18 67.73 • Heart statlog 77.19 80.04 ◦ 79.70 ◦ 81.89 ◦ Hepatitis 76.86 71.11 • 77.49 71.62 • Horse colic 75.79 78.85 ◦ 79.53 ◦ 79.39 ◦ Ionosphere 83.10 87.30 ◦ 88.47 ◦ 87.76 ◦ Kr-vs-kp 59.67 61.53 ◦ 75.80 ◦ 62.72 ◦ Labor 79.93 76.17 • 83.30 ◦ 80.20 Pima diabetes 70.79 70.36 70.29 72.45 ◦ Sick 88.14 84.35 • 93.83 ◦ 84.49 • Sonar 73.29 74.42 74.42 74.93 Vote 91.28 92.71 ◦ 94.32 93.37 ◦ ◦, • statistically significant improvement or degradation
4.2
Y2B 64.43 96.61 83.23 ◦ 70.09 ◦ 81.37 ◦ 75.97 79.91 ◦ 89.57 ◦ 78.06 ◦ 87.10 ◦ 72.84 ◦ 93.98 ◦ 74.75 94.89
The Significance of the Choice of Algorithms and Projection Sets
There are at least two open problems related to projection learning: how to choose a good projection set and which algorithms should be used as learners A and B. To evaluate the significance of the projection sets, we applied algorithm Y also with single-attribute projections (this variant is called Y1 ). Algorithms Y1 and Y2 use Winnow as the learning algorithm at both levels of projection learning, as suggested by Valiant [20], while Y1B and Y2B have Balanced Winnow in the role of the algorithm A. From Table 4 it can be seen that also single-attribute projections increase the prediction accuracy. Using quadratic projections is only somewhat better choice, even if its expressive power is much higher. However, using Balanced Winnow as algorithm A results in significantly better classification accuracies than using basic Winnow. The reason for this is yet to be analyzed. On the other hand, using Balanced Winnow as the algorithm B gives much worse results. This might be due to its ability to express negative correlation of the attributes with the class value. The mistakes of the projection hypotheses can be thought of as noise in the meta-instance z, but in general the prediction of a projection hypothesis does not correlate negatively with the positive class value. Thus, the update rule of Balanced Winnow’s two weight vectors causes oscillation. 4.3
Practical Significance of Attribute-Efficiency
To test how important it is in our test domains that the attribute-efficient Winnow learns the projection hypotheses, we incorporated the basic Perceptron to the role of the learning algorithm A in Y. By Y2P we denote this algorithm combination when quadratic projections are used.
136
T. Elomaa and J.T. Lindgren
Table 5. The effect of using Perceptron instead of Winnow in learning projection hypotheses. Dataset W BW j48 66.87 68.78 73.87 ◦ Breast cancer Breast Wisconsin 95.17 90.45 • 94.69 Credit rating 77.12 80.26 ◦ 86.17 ◦ German credit 69.09 70.53 ◦ 71.54 ◦ Heart statlog 77.19 69.74 • 81.56 ◦ Hepatitis 76.86 79.67 ◦ 78.65 Horse colic 75.79 76.22 85.25 ◦ Ionosphere 83.10 84.25 92.74 ◦ Kr-vs-kp 59.67 73.62 ◦ 99.40 ◦ Labor 79.93 85.47 ◦ 78.43 Pima diabetes 70.79 64.31 • 74.53 ◦ Sick 88.14 95.18 ◦ 97.79 ◦ Sonar 73.29 66.43 • 75.23 Vote 91.28 92.12 96.46 ◦ ◦, • statistically significant improvement or
NB Y2B 73.12 ◦ 64.43 97.25 96.61 85.72 ◦ 83.23 ◦ 74.22 ◦ 70.09 ◦ 83.00 ◦ 81.37 ◦ 83.71 ◦ 75.97 79.23 ◦ 79.91 ◦ 90.43 ◦ 89.57 ◦ 87.84 ◦ 78.06 ◦ 89.07 ◦ 87.10 ◦ 75.03 ◦ 72.84 ◦ 96.37 ◦ 93.98 ◦ 76.87 ◦ 74.75 90.19 94.89 degradation
Y2P 65.64 96.53 82.39 70.20 81.41 76.59 78.74 89.92 89.38 87.60 72.72 94.22 75.71 94.23
◦ ◦ ◦ ◦ ◦ ◦ ◦ ◦
Table 5 gives the results of this experiment. In addition to the really interesting results of Y2B and Y2P , we also list the ones for Winnow, Balanced Winnow, j48, and Na¨ive Bayes for comparison. From these results it is immediately obvious that attribute-efficiency of projection hypothesis learners is not extremely important on our test domains. More or less similar results are obtained using either linear learner to obtain the projection hypotheses. Comparing Y2B and Y2P with each other, both record two statistically significant wins and losses. Y2P is better on domains Kr-vs-kp and Sonar, but loses on Horse colic and Credit rating. Out of these differences only the one on Kr-vs-kp is clearly outstanding. We may conclude that for our test domains Balanced Winnow and Perceptron are equally good projection hypothesis learners. However, let us point out that Perceptron — like Balanced Winnow — does not suit to be used in learning the meta-level concept. The fact that both Balanced Winnow and Perceptron are clearly better suited as projection hypotheses learners than the positive Winnow makes us speculate that, when using simple projection sets, it is the expressive power of the subclassifier that is important. Both Balanced Winnow and Perceptron can have a negative weight for an attribute. Thus, their possible classifiers are a superset of those of positive Winnow. We have not yet tried to verify whether this holds more generally or not. 4.4
Projections versus Input Space Extension
Quadratic projections provide the projection hypotheses with examples in which the respective pair of attributes is true. Adding the pairwise conjunctions of the original attributes to the input space gives the linear learner ability to use the same information. The input space extension, in this case, gives much better time
Experiments with Projection Learning
137
Table 6. Winnow using feature space extended by quadratic variables versus the projection methods. Dataset W BW 66.87 68.78 Breast cancer Breast Wisconsin 95.17 90.45 • Credit rating 77.12 80.26 ◦ German credit 69.09 70.53 ◦ Heart statlog 77.19 69.74 • Hepatitis 76.86 79.67 ◦ Horse colic 75.79 76.22 Ionosphere 83.10 84.25 Kr-vs-kp 79.93 85.47 ◦ Pima diabetes 70.79 64.31 • Sick 88.14 95.18 ◦ Sonar 73.29 66.43 • Vote 91.28 92.12 ◦, • statistically significant
W+C 2 BW+C 2 Y1B 66.59 67.19 68.67 95.57 93.42 • 95.63 79.81 ◦ 80.62 ◦ 82.46 69.78 69.79 70.18 79.52 ◦ 74.44 79.70 76.51 79.58 ◦ 77.49 77.98 78.55 79.53 87.15 ◦ 87.27 ◦ 88.47 74.13 • 88.03 ◦ 83.30 69.88 68.88 • 70.29 89.86 94.47 ◦ 93.83 74.05 71.24 74.42 92.84 92.70 94.32 improvement or degradation
◦ ◦ ◦ ◦ ◦ ◦ ◦
Y2B 64.43 96.61 83.23 70.09 81.37 75.97 79.91 89.57 87.10 72.84 93.98 74.75 94.89
◦ ◦ ◦ ◦ ◦ ◦ ◦ ◦
and space complexities than using projections. However, using higher-order conjunctions would lead the dependence of the computational cost on n to surpass that of projection learning [20]. Table 6 relates Winnow on the original attributes to having its feature space extended by pairwise conjunctions (+C 2 ) and algorithm Y using single-attribute and quadratic projections. It is evident that both methods enhance the basic algorithms’ accuracy, but projection learning, with only a few exceptions, achieves better results. Based on this, both methods seem feasible alternatives to replace Winnow on some task where it is currently used, if one is willing to pay the additional costs in time and space complexity. Input space extension does not seem a particularly good choice for decision tree learning and Na¨ive Bayes. In the problems that we evaluated, the addition of pairwise conjunctions to the input space caused j48 and NB generally to find worse hypotheses. In particular, with an extended input space j48 lost four times to the normal j48 and won only twice, all of these differences statistically significant. For Na¨ive Bayes the ratio was 5:1. The reason for this degradation is the redundancy of the new attributes. Clearly, j48 is already able to represent conjunctions by its hypothesis class. For Na¨ive Bayes the new variables are not independent of the original variables, thus violating the basic assumption of Bayesian learning. 4.5
The Running Time of Projection Learning
Let us finally reflect upon the observed time consumption of our learning algorithms. We know that in the worst case projection learning can be extremely inefficient. How does it fare on these “real-world” data sets? The two-level projection learning cannot, of course, be as efficient as running the linear learners
138
T. Elomaa and J.T. Lindgren Table 7. Time consumption of our test algorithms (in seconds).
Dataset Breast cancer Breast Wisconsin Credit rating German credit Heart statlog Hepatitis Horse colic Ionosphere Kr-vs-kp Labor Pima diabetes Sick Sonar Vote
W 0.08 0.26 0.28 0.44 0.07 0.03 0.17 0.70 1.17 0.01 0.22 1.29 0.30 0.08
BW 0.07 0.25 0.29 0.44 0.07 0.03 0.17 0.68 1.18 0.01 0.22 1.26 0.30 0.08
j48 0.12 0.23 0.44 0.92 0.15 0.10 0.30 0.77 3.66 0.02 0.34 4.37 0.55 0.20
NB 0.02 0.14 0.15 0.26 0.06 0.03 0.07 0.53 0.88 0.01 0.15 1.03 0.30 0.04
Y2B 2.01 1.22 2.13 4.82 0.23 0.38 3.58 20.33 29.23 0.49 0.42 3.77 3.16 0.41
alone. Top-down induction of decision trees and Na¨ive Bayes are also known to be fast learning approaches. Table 7 lists the average execution times of the learning algorithms in one fold of ten-fold cross-validation. Moreover, the averages have been recorded from five repetitions of the ten times repeated ten-fold cross-validation. The experiments have been run on a laptop computer with a slow 350 MHz processor. The two on-line linear learners cannot be separated from each other in execution efficiency. Na¨ive Bayes is even more efficient and decision tree learning by j48 requires somewhat more time than Winnow and Balanced Winnow do. The average time consumption of Y2B is mostly an order of magnitude more than that of NB. The results for Y2P are very similar to those of Y2B and will, thus, be omitted here. In sum, by using projection learning, we can raise the classification accuracy of the straightforward linear learners much closer to that of decision trees and Na¨ive Bayes, but the cost is greatly increased running time and space complexity.
5
Conclusion
Straightforward input space extension by, e.g., conjunctions of attribute pairs can enhance the performance of linear classifiers such as Winnow. However, even better results can be obtained using the provably attribute-efficient classifier combination approach of projection learning. Using Balanced Winnow or Perceptron to learn the projection hypotheses is useful. However, in light of our (limited) experiments, neither should not be used as the learning algorithm for the hypothesis of the projective disjunction. The expense of using projection learning is increased time and space requirement. The increase can at times be so large that it may even become prohibitive
Experiments with Projection Learning
139
in realistic problems involving tens of thousands of attributes. In our experiments the overall performance of projection learning was observed to approach that of decision tree learning and Na¨ive Bayes. However, projection learning is significantly more inefficient than either of these learning algorithms. We set out to test projection learning on “real-world” data. However, as our experiment with using Perceptron to learn the projection hypotheses demonstrates, attribute-efficiency is not required here. It would be interesting to see the same experiments repeated for domains where attribute-efficiency of the learner makes a difference.
References 1. Blake, C.L., Merz, C.J.: UCI repository of machine learning databases. Department of Information and Computer Science, University of California, Irvine (1998). http://www.ics.uci.edu/∼mlearn/MLRepository.html 2. Blum, A.: Empirical support for Winnow and weighted-majority based algorithms: results on a calendar scheduling domain. Proc. Twelfth International Conference on Machine Learning. Morgan Kaufmann, San Francisco, CA (1995) 64–72 3. Duda, R.O., Hart, P.E., Stork, D.G.: Pattern Classification. Second Edition. John Wiley and Sons, New York, NY (2000) 4. Fayyad, U.M., Irani, K.B.: Multi-interval discretization of continuous-valued attributes for classification learning. Proc. Thirteenth International Joint Conference on Artificial Intelligence. Morgan Kaufmann, San Francisco, CA (1993) 1022–1027 5. Golding, A.R., Roth, D.: A Winnow-based approach to context-sensitive spelling correction. Mach. Learn. 34 (1999) 107–130 6. Haussler, D. Quantifying inductive bias: AI learning algorithms and Valiant’s learning framework. Artif. Intell. 36 (1988) 177–222 7. Haykin, S.: Neural Networks: A Comprehensive Foundation. Second Edition. Prentice Hall, Upper Saddle River, NJ (1999) 8. Khardon, R., Roth, D., Valiant L.G.: Relational learning for NLP using linear threshold elements. Proc. Sixteenth International Joint Conference on Artificial Intelligence. Morgan Kaufmann, San Francisco, CA (1999) 911–919 9. Kivinen, J., Warmuth, M.K., Auer, P.: The Perceptron algorithm versus Winnow: linear versus logarithmic mistake bounds when few inputs are relevant. Artif. Intell. 97 (1997) 325–343 10. Littlestone, N.: Learning quickly when irrelevant attributes are abound: a new linear threshold algorithm. Mach. Learn. 2 (1988) 285–318 11. Littlestone, N.: Mistake bounds and logarithmic linear-threshold learning algorithms. Ph.D. Thesis, Report UCSC-CRL-89-11, University of California, Santa Cruz (1989) 12. Quinlan, J.R.: C4.5: Programs for Machine Learning. Morgan Kaufmann, San Mateo, CA (1993) 13. Rosenblatt, F.: The Perceptron: a probabilistic model for information storage and organization in the brain. Psychological Rev. 65 (1958) 386–407 14. Roth, D., Yang, M.-H., Ahuja, N.: Learning to recognize three-dimensional objects. Neural Comput. 14 (2002) 1071–1103 15. Servedio, R.A.: On PAC learning using Winnow, Perceptron, and a Perceptron-like algorithm. Proc. Twelfth Annual Conference on Computational Learning Theory. (1999) 296–307
140
T. Elomaa and J.T. Lindgren
16. Servedio, R.A.: Computational sample complexity and attribute-efficient learning. J. Comput. Syst. Sci. 60 (2000) 161–178 17. Ting, K.M., Witten, I.H.: Stacked generalizations: when does it work? Proc. Fifteenth International Joint Conference on Artificial Intelligence. Morgan Kaufmann, San Francisco, CA (1997) 866–873 18. Uehara, R., Tsuchida, K., Wegener, I.: Identification of partial disjunction, parity, and threshold functions. Theor. Comput. Sci. 230 (2000) 131–147 19. Valiant, L.G.: Circuits of the Mind. Oxford University Press, Oxford (1994) 20. Valiant, L.G.: Projection learning. Mach. Learn. 37 (1999) 115–130 21. Valiant, L.G.: A neuroidal architecture for cognitive computation. J. ACM 47 (2000) 854–882 22. Vapnik, V.N.: Statistical Learning Theory. John Wiley and Sons, New York, NY (1998) 23. Witten, I.H., Frank, E.: Data Mining: Practical Machine Learning Tools and Techniques with Java Implementations. Morgan Kaufmann, San Francisco, CA (2000) 24. Wolpert, D.H.: Stacked generalization. Neural Networks 5 (1992) 241–259
Improved Dataset Characterisation for Meta-learning 1
1
2
2
Yonghong Peng , Peter A. Flach , Carlos Soares , and Pavel Brazdil 1
Department of Computer Science, University of Bristol, UK {yh.peng,peter.flach}@bristol.ac.uk 2 LIACC/Fac. of Economics, University of Porto, Portugal {csoares,pbrazdil}@liacc.up.pt
Abstract. This paper presents new measures, based on the induced decision tree, to characterise datasets for meta-learning in order to select appropriate learning algorithms. The main idea is to capture the characteristics of dataset from the structural shape and size of decision tree induced from the dataset. Totally 15 measures are proposed to describe the structure of a decision tree. Their effectiveness is illustrated through extensive experiments, by comparing to the results obtained by the existing data characteristics techniques, including data characteristics tool (DCT) that is the most wide used technique in metalearning, and Landmarking that is the most recently developed method.
1 Introduction Extensive research has been performed to develop appropriate machine learning techniques for different data mining problems, and has led to a proliferation of different learning algorithms. However, previous work has shown that no learner is generally better than another learner. If a learner performs better than another learner on some learning situations, then the first learner must perform worse than the second learner on other situations [18]. In other words, no single learning algorithm can perform well and uniformly outperform other algorithms over all data mining tasks. This has been confirmed by the ‘no free lunch theorems’ [29,30]. The major reasons are that a learning algorithm has different performance in processing different dataset and that different learning algorithms are implemented with different search heuristics, which results in variety of ‘inductive bias’ [15]. In real-world applications, the users need to select an appropriate learning algorithm according to the mining task that he is going to perform [17,18,1,7,20,12]. An inappropriate selection of algorithm will result in slow convergence, or even lead a sub-optimal local minimum. Meta-learning has been proposed to deal with the issues of algorithm selection [5, 8]. One of the aims of meta-learning is to assist the user to determine the most suitable learning algorithm(s) for the problem at hand. The task of meta-learning is to find functions that map datasets to predicted data mining performance (e.g., predictive accuracies, execution time, etc.). To this end meta-learning uses a set of attributes, called meta-attributes, to represent the characteristics of data mining tasks, and search for the correlations between these attributes and the performance of learning algorithms [5,10,12]. Instead of executing all learning algorithms to obtain the optimal one, meta-learning is performed on the meta-data characterising the data mining tasks. S. Lange, K. Satoh, and C.H. Smith (Eds.): DS 2002, LNCS 2534, pp. 141-152, 2002. © Springer-Verlag Berlin Heidelberg 2002
142
Y. Peng et al.
The effectiveness of meta-learning is largely dependent on the description of tasks (i.e., meta-attributes). Several techniques have been developed, such as data characterisation techniques (DCT) [13] to describe the problem to be analyzed, including simple measures (e.g. the number of attributes, the number of classes et al.), statistical measures (e.g. mean and variance of numerical attributes), and information theorybased measures (e.g. entropy of classes and attributes). There is, however, still a need for improving the effectiveness of meta-learning by developing more predictive metaattributes and selecting the most informative ones [9]. The aim of this work is to investigate new methods to characterize the dataset for meta-learning. Previously, Bensusan et al’s proposed to capture the information from the induced decision trees for characterizing the learning complexity [3, 32]. In [3], they listed 10 measures based on the decision tree, such as the ratio of the number of nodes to the number of the attributes, the ratio of number of nodes to the number of training instances; however, they did evaluate the performance of these measures. In our recent work, we have re-analysed the characteristics of decision trees, and proposed 15 new measures, which focus on characterizing the structural properties of decision tree, e.g., the number of nodes and leave, the statistical measures regarding the distributes of nodes at each level and along each branch, the width and depth of the tree, the distribution of attributes in the induced decision tree. These measure have been applied to rank 10 learning algorithms. The experimental results show the enhancement of performance in ranking algorithms, compared to the DCT, which is the commonly used technique, and landmarking, a recently introduced technique [19,2]. This paper is organized as following. In section 2, some related work is introduced, including meta-learning methods for learning algorithm selection and data characterisation. The proposed method for characterising datasets is stated in detail in section 3. Experiments illustrating the effectiveness of the proposed method are described in section 4. Section 5 concludes the paper, and points out interesting possibilities for future work.
2 Related Work There are two basic tasks in meta-learning: the description of the learning tasks (datasets), and the correlation between the task description and the optimal learning algorithm. The first task is to characterise datasets with meta-attributes, which constitutes the meta-data for meta-learning, whilst the second is the learning at meta-level, which develops the meta-knowledge for selecting appropriate algorithm in classification. For algorithm selection, several meta-learning strategies have been proposed [6,25,26]. In general, there are three options concerning the target. One is to select the best learning algorithm, i.e. to select the algorithm that is expected to produce the best model for the task. The second is to select a group of learning algorithms including not only the best algorithm but also the algorithms that are not significantly worse than the best one. The third possibility is to rank the learning algorithms according to their predicted performance. The ranking will assist the user to finally select the learning algorithm. This ranking-based meta-learning is the main approach in the Esprit Project MetaL (www.metal-kdd.org).
Improved Dataset Characterisation for Meta-learning
143
Ranking the preference order of algorithms is performed based on estimating their performance in mining the associating dataset. In data mining, performance can be measured not only in term of accuracy but also time or understandability of model. In this paper, we assess performance with the Adjusted Ratio of Ratios (ARR) measure, which combines the accuracy and time. ARR gives a measure regarding the advantage of a learning algorithm over another algorithm in terms of their accuracy and the execution time. The user can adjust the importance of accuracy relative to time by a tunable parameter. The ‘zoomed ranking’ method proposed by Soares [26] based on ARR is used in this paper for algorithm selection, taking into account of accuracy and execution time simultaneously. The first attempt to characterise datasets in order to predict the performance of classification algorithm was done by Rendell et al. [23]. So far, two main strategies have been developed in order to characterise a dataset for suggesting which algorithm is more appropriate for a specific dataset. First one is the technique that describes the properties of datasets using statistical and informational measures. In the second one a dataset is characterised using the performance (e.g. accuracy) of a set of simplified learners, which was called landmarking [19,2]. The description of a dataset in terms of its information/statistical properties appeared for the first time within the framework of the STATLOG project [14]. The authors used a set of 15 characteristics, spanning from simple ones, like the number of attributes or the number of examples, to more complex ones, such as the first canonical correlation between the attributes and the class. This set of characteristics was later applied in various studies for solving the problem of algorithm selection [5,28,27]. They distinguish three categories of dataset characteristics, namely simple, statistical and information theory based measures. Statistical characteristics are mainly appropriate for continuous attributes, while information theory based measures are more appropriate for discrete attributes. Linder and Studer [13] provide an extensive list of information and statistical measures of a dataset. They provide a tool for the automatic computation of these characteristics, which was called Data Characterisation Tool (DCT). Sohn [27] also uses the STATLOG set as a starting point. After careful evaluation of their properties in a statistical framework, she noticed that some of the characteristics are highly correlated, and she omitted the redundant ones in her study. Furthermore, she introduces new features that are transformation or combinations of these existing measures, like ratios or seconds powers [27]. An alternative approach to characterise datasets called landmarking was proposed in [19,2]. The intuitive idea behind landmarking is that the performance of simple learner, called landmarker, can be used to predict the performance of given candidate algorithms. That is, given landmarker A and B, if we know landmarker A outperforms landmarker B on the present task, then we could select the learning algorithms that has the same inductive bias of landmarker A to perform this data mining task. It has to be ensured that the chosen landmarkers have quite distinct learning biases. As a closely related approach, Bensusan et al. had also proposed to use the information computed from the induced decision trees to characterize learning tasks [3, 32]. They listed 10 measures based on the unpruned tree but did not evaluate their performance.
144
Y. Peng et al.
3 The Proposed Measures for Describing Data Characteristics The task of characterizing dataset for meta-learning is to capture the information about learning complexity on the given dataset. This information should enable the prediction of performance of learning algorithms. It should also be computable within a relative short time comparing to the whole learning process. In this section we introduce new measures to characterize the dataset by measuring a variety of properties of a decision tree induced from that dataset. The major idea here is to measure the model complexity by measuring the structure and size of decision tree, and use these measures to predict the complexity of other learning algorithms. We employed the standard decision tree learner, c5.0tree. There are several reasons for selecting decision trees. The major reason is that decision tree has been one of the most popularly used machine learning algorithms, and the induction of decision tree is deterministic, i.e. the same training set could produce the similar structure of decision tree. Definition. A standard tree induced with c5.0 (or possibly ID3 or c4.5) consists of a number of branches, one root, a number of nodes and a number of leaves. One branch is a chain of nodes from root to a leaf; and each node involves one attribute. The occurrence of an attribute in a tree provides the information about the importance of the associated attribute. The tree width is defined as the number of lengthways partitions divided by parallel nodes or leave from the leftmost to the rightmost nodes or leave. The tree level is defined as the breadth-wise partition of tree at each success branches, and the tree height is defined by the number of tree levels, as shown in Fig.1. The length of a branch is defined as the number of nodes in the branch minus one. level-1
F
level-3 level-4
- root x2 c1
x1 c3
c1
TreeHigh
level-2
x1
- node - leaf x1,x2 - attributes c1,c2,c3 - classes
TreeWidth
Fig. 1. Structure of Decision Tree.
We propose, based on above notations, to describe decision tree in term of the following three aspects: a) outer-profile of tree; b) statistic for intra-structure: including tree levels and branches; c) statistic for tree elements: including nodes and attributes. To describe the outer-profile of the tree, the width of tree (treewidth) and the height of the tree (treeheight) are measured according to the number of nodes in each level and the number of levels, as illustrated in Fig.1. Also, the number of nodes (NoNode) and the number of leaves (NoLeave) are used to describe the overall property of a tree. In order to describe the intra-structure of the tree, the number of nodes at each level and the length of each branch are counted. Let us represent them with two vectors
Improved Dataset Characterisation for Meta-learning
145
denoted as NoinL=[v1,v2,…vl] and LofB=[L1,L2,….Lb] respectively, where vi is the number of nodes at the ith level, Lj is the length of jth branch, l and b is the number of levels (treeheight) and number of branches. Based on NoinL and LofB, the following four measures can be generated: The maximum and minimum number of nodes at one level:
maxLevel = max(v1,v 2 ,...v l ) minLevel = min(v1,v 2 ,...v l )
(1)
(As the minLevel is always equal to 1, it is not used.) The mean and standard deviation of the number of nodes on levels:
(∑ v ) l , l
meanLevel =
devLevel =
∑
l i =1
i =1 i
(2)
(vi − meanLevel ) 2 (l − 1)
The length of longest and shortest branches:
LongBranch = max(L1 , L2 ,...Lb )
(3)
ShortBranch = min(L1 , L2 ,...Lb ) The mean and standard deviation of the branch lengths:
(∑
b
meanBranch = devBranch =
∑
b j =1
j =1
)
Lj b ,
(4)
( L j − meanBranch) 2 (b − 1)
Besides the distribution of nodes, the frequency of attributes used in a tree provides useful information regarding the importance of each attribute. The times of each attribute is used in a tree represents by a vector NoAtt=[nAtt1, nAtt2, …. nAttm], where nAttk is the number of times the kth attribute is used and m is the total number of attributes in the tree. Again, the following measures are used: The maximum and minimum occurrence of attributes:
maxAtt = max(nAtt1,nAtt2 ,...nAttmm )
(5)
minAtt = min(nAtt1,nAtt2 ,...nAttmm ) Mean and standard deviation of the number of occurrences of attributes:
meanAtt =
devAtt =
∑
m
i =1
(∑
m
i =1
)
nAtt i m ,
(6)
(nAtti − meanAtt) 2 (m − 1)
As a result, a total of 15 meta-attributes (i.e., treewidth, treeheight, NoNode, NoLeave, maxLevel, meanLevel, devLevel, LongBranch, ShortBranch, meanBranch, devBranch, maxAtt, minAtt, meanAtt, devAtt) has been defined.
146
Y. Peng et al.
4 Experimental Evaluation In this section we experimentally evaluate the proposed data characteristics. In section 4.1 we describe our experimental set-up, in section 4.2 we compare our proposed meta-features with DCT and landmarking, and in section 4.3 we study the effect of meta-feature selection. 4.1 Experimental Set-up The meta-learning technique employed in this paper is an instance-based learning algorithm based ranking. Given a data mining problem (a dataset to analyze), the kNearest Neighbor (kNN) algorithm is used to select a subset with k dataset, whose characteristics are similar to the characteristics of the present dataset according to some distance function, from the benchmark datasets. Next, a ranking of their preference according to the the selected datasets is generated based on the adjusted ratio of ratios (ARR), a multicriteria evaluation measure that combines the accuracy and time. ARR has a parameter to enable the user to adjust the relative importance of accuracy and time according to his particular data mining objective. More details can be found in [26]. To evaluate a recommended ranking, we calculate its similarity to an ideal ranking obtained for the same dataset. The ideal ranking is obtained by estimating the performance of the candidate algorithms using 10-fold cross-validation. Similarity is measured using the Spearman’s rank correlation coefficient [29]. n n 6D2 2 2 2 (7) rs = 1− ,D = ∑i=1 Di = ∑i=1 (ri − ri ) 2 n(n − 1) where the
ri and r i are the predicted ranking and actual ranking for algorithm i re-
spectively. The bigger rs is, better of ranking result is, with rs = 1 if the ranking is same as the ideal ranking. In our experiments, a total of 10 learning algorithms, including c5.0tree, c5.0boost and c5.0rules [21], Linear Tree (ltree), linear discriminant (lindiscr), MLC++ Naive Bayes classifier (mlcnb) and Instance-based leaner (mlcib1) [11], Clementine Multilayer Perceptron (clemMLP), Clementine Radial Basis Function (clemRBFN) and rule learner (ripper), have been evaluated on 47 datasets, which are mainly from the UCI repository [4]. The error rate and time were estimated using 10-fold cross-validation. The leave-one-out method is used to evaluate the performance of ranking, i.e., the performance for ranking the 10 given learning algorithms for each dataset on the basis of the other 46 datasets. 4.2 Comparison with DCT and Landmarking The effect of new proposed meta-attributes (called DecT) has been evaluated on ranking of these 10 learning algorithms. In this section, we compare the ranking performance generated by DecT (15 meta-attributes) to that generated by DCT (25 meta-
Improved Dataset Characterisation for Meta-learning
147
attributes) and Landmarking (5 meta-attributes). The used 25 DCT and 5 Landmarking meta-attributes are listed in the Appendix. The first experiment is performed to rank the given 10 learning algorithms on the 47 datasets, in which, the parameters k=10 (meaning the 10 most similar datasets are first selected from the 46 datasets in kNN algorithm), Kt=100 (meaning that we are willing to tread 1% in accuracy for a 100 times speed-up or slowdown [26]) is used. The ranking performance is measured with rs (Eq. (7)). The results of ranking performance of using DCT, landmarking and DecT are shown in Fig. 2. The overall average performance for DCT, Landmarking and DecT are 0.613875, 0.634945 and 0.676028 respectively, which demonstrates the improvement of DecT in ranking algorithms, comparing to DCT and Landmarking.
'&7
/DQGPDUNLQJ
'HF7
Fig. 2. Ranking performance for 47 datasets using DCT, landmarking and DecT.
In order to look in more detail at the improvement of DecT over DCT and Landmarking, we performed the experiment of ranking using different values of k and Kt. As stated in [26], the parameter Kt represents the relative importance of accuracy and execution time in selecting the learning algorithm (i.e., higher Kt means the accuracy is more important and time is less important in algorithm selection). Fig. 3 shows that for Kt={10,100,1000}, using DecT improves the performance comparing with the use of DCT and landmarking meta-attributes. NW
NW '&7
/DQGPDUNLQJ
NW 'HF7
Fig. 3. The ranking performance for different values of Kt.
148
Y. Peng et al.
Fig. 4 shows the performance of ranking based on different zooming degree (different k), i.e., selecting different number of similar datasets, based on which the ranking is performed. From these results, we observe that 1) for all different values of k, DecT produces better ranking performance than DCT and landmarking; 2) best performance is obtained by selecting 10-25 datasets among 47 datasets. N
N
N
N
N
'&7
N
N
N
N
/DQGPDUNLQJ
N
'HF7
Fig. 4. The ranking performance for different values of k.
4.3 Performing Meta-feature Selection The kNN-nearest neighbor (kNN) method, employed to select k datasets for ranking the performance of learning algorithms for the given dataset, is known to be sensitive to the irrelevant and redundant features. Using smaller number of features could help to improve the performances of kNN, as well as to reducing the time used in metalearning. In our experiments, we manually reduced the number of DCT meta-features from 25 to 15 and 8, and compare their results to those obtained based on the same number of DecT meta-features. The reduction for DCT meta-features is performed by removing the features thought to be redundant, and the features having a lot of nonappl (missing or error) values, and the reduction for DecT meta-features are performed by removing redundant features that are highly correlated. N
N
N
N
N
N
N
'&7
'&7
'HF7
'HF7
N
N
'&7
Fig. 5. Results for reduced meta-features.
N
Improved Dataset Characterisation for Meta-learning
149
The ranking performances for these reduced meta-features are shown in Fig.5, in which DCT(8), DCT(15), DecT(8) represent the reduced 8, 15 DCT meta-features and 8 DecT meta-features, DCT(25) and DecT(15) represent the full DCT and DecT metafeatures respectively. From Fig.5, we can observe that feature selection did not significantly influence the performance of either DCT or DecT, and that the latter outperforms the former across the board.
5 Conclusions and Future Work Meta-learning strategy, under the framework of MetaL, aims at assisting the user in select appropriate learning algorithm for the particular data mining task. Describing the characteristics of dataset in order for estimating the performance of learning algorithm is the key to develop a successful meta-learning system. In this paper, we proposed new measures to characterise the dataset. The basic idea is to process the dataset using a standard tree induction algorithm, and then to capture the information regarding the dataset’s characteristics from the induced decision tree. The decision tree is generated using standard c5.tree algorithm. A total of 15 measures, which constitute the meta-attributes for meta-learning, have been proposed for describing different kind of properties of a decision tree. The proposed measures have been applied in ranking the learning algorithms based on accuracy and time. Extensive experimental results have illustrated the improvement of ranking performance by using the proposed 15 meta-attributes, compared to the 25 DCT and 5 Landmarking meta-features. In order to avoid the effect of redundant or irrelevant features on the performance of kNN learning, we also compared the performance of ranking based on the selected 15 DCT meta-features and DecT, and selected 8 DCT and DecT meta-features. The results suggest that feature selection does not significantly change performance of either DCT or DecT. In other experiments, we observed that the combination of DCT with DecT or Landmarking with DCT and DecT did not produce better performance than DecT. This is an issue that we are interested in further investigation. The major reason may come from the use of k-nearest neighbor learning in zooming based ranking strategy. One possibility is to test the performance of the combination of DCT, landmarking and DecT in other meta-learning strategies, such as best algorithm selection. Another interesting subject is to look at the change of shape and size of the decision tree along with the change of examples used in tree induction, as it will be useful if it is possible to capture the data characteristics based on sampled dataset. This is especially important for large datasets. Acknowledgements. this work is supported by the MetaL project (ESPRIT Reactive LTR 26.357).
150
Y. Peng et al.
References 1. 2.
3.
4. 5.
6. 7. 8.
9. 10.
11. 12.
13.
14. 15. 16. 17. 18.
C. E. Brodley. Recursive automatic bias selection for classifier construction. Machine Learning, 20:63-94, 1995. H. Bensusan, and C. Giraud-Carrier. Discovering Task Neighbourhoods through Landmark th Learning Performances. In Proceedings of the 4 European Conference on Principles of Data Mining and Knowledge Discovery. 325-330, 2000. H. Bensusan, C. Giraud-Carrier, and C. Kennedy. Higher-order Approach to Metalearning. The ECML’2000 workshop on Meta-Learning: Building Automatic Advice Strategies for Model Selection and Method Combination, 109-117, 2000. C. Blake, E. Keogh, C. Merz. www.ics.uci.edu/~mlearn/mlrepository.html. University of California, Irvine, Dept. of Information and Computer Sciences,1998. P. Brazdil, J. Gama, and R. Henery. Characterizing the Applicability of Classification Algorithms using Meta Level Learning. In Proceeedings of the European Conference on Machine Learning, ECML-94, 83-102, 1994. T. G. Dietterich. Approximate statistical tests for comparing supervised classification learning algorithms. Neural Computation, 10(7):1895-1924, 1998. E. Gordon, and M. desJardin. Evaluation and Selection of Biases. Machine Learning, 20(12):5-22, 1995. A. Kalousis, and M. Hilario. Model Selection via Meta-learning: a Comparative Study. In Proceedings of the 12th International IEEE Conference on Tools with AI, Vancouver. IEEE press. 2000. A. Kalousis, and M. Hilario. Feature Selection for Meta-Learning. In Proceedings of the 5th Pacific Asia Conference on Knowledge Discovery and Data Mining. 2001. C. Koepf, C. Taylor, and J. Keller. Meta-analysis: Data characterisation for classification and regression on a meta-level. In Antony Unwin, Adalbert Wilhelm, and Ulrike Hofmann, editors, Proceedings of the International Symposium on Data Mining and Statistics, Lyon, France, (2000). nd R. Kohavi. Scaling up the Accuracy of Naïve-bayes Classifier: a Decision Tree hybrid. 2 Int. Conf. on Knowledge Discovery and Data Mining, 202-207. (1996) M. Lagoudakis, and M. Littman. Algorithm selection using reinforcement learning. In Proceedings of the Seventeenth International Conference on Machine Learning (ICML2000), 511-518, Stanford, CA. (2000) C. Linder, and R. Studer. AST: Support for Algorithm Selection with a CBR Approach. th Proceedings of the 16 International Conference on Machine Learning, Workshop on Recent Advances in Meta-Learning and Future Work. 1999. D. Michie, D. Spiegelhalter, and C. Toylor. Machine Learning, Neural Network and Statistical Classification. Ellis Horwood Series in Artificial Intelligence. 1994. T. Mitchell. Machine Learning. MacGraw Hill. 1997. S. Salzberg. On Comparing Classifiers: Pitfalls to Avoid and a Recommended Approach. Data Mining and Knowledge Discovery, 1:3, 317-327, 1997. C. Schaffer. Selecting a Claassification Methods by Cross Validation, Machine Learning, 13, 135-143,1993. C. Schaffer. Cross-validation, stacking and bi-level stacking: Meta-methods for classification learning. In P. Cheeseman and R. W. Oldford, editors, Selecting Models from Data: Artificial Intelligence and Statistics IV, 51-59, 1994.
Improved Dataset Characterisation for Meta-learning
151
19. B. Pfahringer, H. Bensusan, and C. Giraud-Carrier. Tell me who can learn you and I can th tell you who you are: Landmarking various Learning Algorithms. Proceedings of the 17 Int. Conf. on Machine Learning. 743-750, 2000. 20. F. Provost, and B. Buchanan. Inductive policy: The pragmatics of bias selection. Machine Learning, 20:35-61, 1995. 21. J. R. Quinlan. C4.5: Programs for Machine Learning, Morgan Kaufman, 1993. 22. J. R. Quinlan. C5.0: An Informal Tutorial, RuleQuest, www.rulequest.com, 1998. 23. L. Rendell, R. Seshu, and D. Tcheng. Layered Concept Learning and Dynamically Varith able Bias Management. 10 Inter. Join Conference on AI. 308-314, 1987. th 24. C. Schaffer. A Conservation Law for Generalization Performance. Proceedings of the 11 International Conference on Machine Learning, 1994. 25. C. Soares. Ranking Classification Algorithms on Past Performance. Master’s Thesis, Faculty of Economics, University of Porto, 2000. 26. C. Soares. Zoomed Ranking: Selection of Classification Algorithms based on Relevant th Performance Information. Proceedings of the 4 European Conference on Principles of Data Mining and Knowledge Discovery, 126-135, 2000. 27. S. Y. Sohn. Meta Analysis of Classification Algorithms for Pattern Recognition. IEEE Trans. on Pattern Analysis and Machine Intelligence, 21, 1137-1144, 1999. 28. L. Todorvoski, and S. Dzeroski. Experiments in Meta-Level Learning with ILP. Proceedth ings of the 3 European Conference on Principles on Data Mining and Knowledge Discovery, 98-106, 1999. 29. A. Webster. Applied Statistics for Business and Economics, Richard D Irwin Inc, 779-784, 1992. 30. D. Wolpert. The lack of a Priori Distinctions between Learning Algorithms. Neural Computation, 8, 1341-1390, 1996. 31. D. Wolpert. The Existence of a Priori Distinctions between Learning Algorithms. Neural Computation, 8, 1391-1420, 1996. 32. H. Bensusan. God doesn’t always shave with Occam’s Razor - learning when and how to prune. In Proceedigs of the 10th European Conference on Machine Learning, 119--
124, Berlin, Germany, 1998.
Appendix DCT Meta-attributes: 1. Nr_attributes: Number of attributes. 2. Nr_sym_attributes: Number of symbolic attributes. 3. Nr_num_attributes: Number of numberical attributes. 4. Nr_examples: Number of records/examples. 5. Nr_classes: Number of classes. 6. Default_accuracy: The default accuracy. 7. MissingValues_Total: Total number of missing values. 8. Lines_with_MissingValues_Total: Number of examples having missing values. 9. MeanSkew: Mean skewness of the numerical attributes. 10. MeanKurtosis: Mean kurtosis of the numerical attributes.
152
Y. Peng et al.
11. NumAttrsWithOutliers: number of attributes for which the ratio between the alpha-trimmed standard-deviation and the standard-deviation is larger than 0.7 12. MStatistic: Boxian M-Statistic to test for equality of covariance matrices of the numerical attributes. 13. MStatDF: Degrees of freedom of the M-Statistic. 14. MStatChiSq: Value of the Chi-Squared distribution. 15. SDRatio: A transformation of the M-Statistic which assesses the information in the co-variance structure of the classes. 16. Fract: Relative proportion of the total discrimination power of the first discriminant function. 17. Cancor1: Canonical correlation of the best linear combination of attributes to distinguish between classes. 18. WilksLambda: Discrimination power between the classes. 19. BartlettStatistic: Bartlett’s V-Statistic to the significance of discriminant functions. 20. ClassEntropy: Entropy of classes. 21. EntropyAttributes: Entropy of symbolic attributes. 22. MutualInformation: Mutual information between symbolic attributes and classes. 23. JointEntropy: Average joint entropy of the symbolic attributes and the classes. 24. Equivalent_nr_of_attrs: ratio between class entropy and average mutual information, providing information about the number of necessary attributes for classification. 25. NoiseSignalRatio: Ratio between noise and signal, indicating the amount of irrelevant information for classification. Landmarking Meta-features: 1. Naive Bayes 2. Linear discriminant 3. Best node of decision tree 4. Worst node of decision tree 5. Average node of decision tree
Racing Committees for Large Datasets Eibe Frank, Geoffrey Holmes, Richard Kirkby, and Mark Hall Department of Computer Science University of Waikato Hamilton, New Zealand {eibe, geoff, rkirkby, mhall}@cs.waikato.ac.nz
Abstract. This paper proposes a method for generating classifiers from large datasets by building a committee of simple base classifiers using a standard boosting algorithm. It permits the processing of large datasets even if the underlying base learning algorithm cannot efficiently do so. The basic idea is to split incoming data into chunks and build a committee based on classifiers built from these individual chunks. Our method extends earlier work by introducing a method for adaptively pruning the committee. This is essential when applying the algorithm in practice because it dramatically reduces the algorithm’s running time and memory consumption. It also makes it possible to efficiently “race” committees corresponding to different chunk sizes. This is important because our empirical results show that the accuracy of the resulting committee can vary significantly with the chunk size. They also show that pruning is indeed crucial to make the method practical for large datasets in terms of running time and memory requirements. Surprisingly, the results demonstrate that pruning can also improve accuracy.
1
Introduction
The ability to process large datasets becomes more and more important as institutions automatically collect data for the purpose of data mining. This paper addresses the problem of generating classification models from large datasets, where the task is to predict the value of a nominal class given a set of attributes. Many popular learning algorithms for classification models are not directly applicable to large datasets because they are too slow and/or require too much memory. Apart from specialized algorithms for particular classification models, several generic remedies for the above problems have been proposed in the literature. They can be broadly classified into subsampling strategies [8,13] and learning using committee machines [11,3,4,12,14]. Of these two strategies, the latter one appears to be particularly promising because (a) it does not require any data to be discarded when building the classifier, and (b) it allows for incremental learning because the model can be updated when a new chunk of data arrives. The basic idea of committee-based learning for large datasets is to build a committee by splitting the data into chunks, learning a model from each chunk, S. Lange, K. Satoh, and C.H. Smith (Eds.): DS 2002, LNCS 2534, pp. 153–164, 2002. c Springer-Verlag Berlin Heidelberg 2002 ÿ
154
E. Frank et al.
and combining the predictions of the different models to form an overall prediction. If the maximum chunk size is kept small, polynomial time algorithms can be applied to induce the individual models in a reasonable amount of time. Working with chunks also makes the process memory-efficient because a chunk can be discarded once it has been processed by the learning scheme. In this paper we focus on using a boosting algorithm for building the committee machines. Boosting has the advantage that it can combine “weak” classifiers into a committee that is significantly more powerful than each individual classifier [5]. This is particularly advantageous in our application because the individual classifiers are built from relatively small samples of data and are therefore necessarily “weak.” The idea of using boosting for large datasets is not new, and appears to have been proposed first by Breiman [3]. The main contribution of this paper is a method for adaptively and efficiently pruning the incrementally built committee of classifiers, which makes the process computationally feasible for large datasets. It also makes it possible to choose an appropriate chunk size among several candidates based on “racing” the candidate solutions. This is important because the correct chunk size cannot be determined a priori. Apart from making the method practical, pruning also has the (desirable) side-effect that the resulting predictions can become more accurate. This paper is structured as follows. In Section 2 we present our method for constructing committees on large datasets. We start with a naive method that does not perform any pruning and then move on to a more practical method that incorporates a pruning strategy. Finally, we discuss how the resulting committees are raced. Section 3 contains experimental results on a collection of benchmark datasets, demonstrating the importance of choosing an appropriate chunk size and using a pruning strategy. Section 4 discusses related work on combining classifiers built from chunks of data. Section 5 summarizes the contributions made in this paper.
2
The Algorithm
We first describe the basic algorithm—called “incremental boosting”—that generates the committee from incoming chunks of data. Then we explain how incremental boosting can be modified to incorporate pruning. Finally we present the racing strategy for pruned committees built from different chunk sizes. 2.1
Incremental Boosting
Standard boosting algorithms implement the following basic strategy. In the first step a prediction model is built from the training data using the underlying “weak” learning algorithm and added to the (initially empty) committee. In the second step the weight associated with each training instance is modified. This two-step process is repeated for a given number of iterations. The resulting
Racing Committees for Large Datasets
155
committee is used for prediction by combining the predictions of the individual models. Boosting works well because the individual models complement each other. After a model has been added to the committee, the instances’ weights are changed so that instances that the committee finds “difficult” to classify correctly get a high weight, and those that are “easy” to classify get a low weight. The next model that is built will then focus on the difficult parts of the instance space instead of the easy ones (where difficulty is measured according to the committee built so far). This strategy generates a diverse committee of models, and this diversity appears to be the main reason why boosting works so well in practice. It turns out that boosting can also be viewed as a statistical estimation procedure called “additive logistic regression.” LogitBoost [6] is a boosting procedure that is a direct implementation of an additive logistic regression method for maximizing the multinomial likelihood of the data given the committee. In contrast to AdaBoost and related algorithms it has the advantage that it is directly applicable to multiclass problems. It jointly optimizes the class probability estimates for the different classes and appears to be more accurate than other multi-class boosting methods [6]. For this reason we chose it as the underlying boosting mechanism for our incremental boosting strategy. LogitBoost assumes that the underlying weak learner is a regression algorithm that attempts to minimize the mean-squared error. This can, for example, be a regression tree learner. For the experimental results reported in this paper we used a learning algorithm for “regression stumps.” Regression stumps are 1-level regression trees. In our implementation these stumps have ternary splits where one branch handles missing attribute values. The only difference between standard boosting and incremental boosting is that the latter uses a different dataset in each iteration of the boosting algorithm: the incoming training data is split into mutually exclusive “chunks” of the same size and a model is generated for each of these chunks. When a new chunk of data becomes available the existing committee’s predictions for this chunk are used to weight the data and a new model is learned on the weighted chunk and added to the committee. In this fashion a committee of boosted models is incrementally constructed as training data is processed. Figure 1 depicts the basic algorithm for incremental boosting. This algorithm assumes that new models can be added to the committee until the data is exhausted. This may not be feasible because of memory constraints. In the next section we will discuss a pruning method for reducing the committee’s size. Another drawback of the algorithm is its time complexity, which is quadratic in the number of chunks (and therefore quadratic in the number of training instances). In each iteration i, i − 1 base models are invoked in order to make predictions for chunk Ki (so that its instances can be weighted). Consequently this naive algorithm can only be applied if the chunk size is large relative to the size of the full dataset.
156
E. Frank et al. START with an empty committee K0 REPEAT FOR next data chunk Ci DO BEGIN IF (i > 1) THEN weight chunk Ci according to the predictions of K0..i−1 learn model Mi for chunk Ci and add to committee K0..i−1 END UNTIL no more chunks Fig. 1. Incremental boosting.
2.2
Incremental Boosting with Pruning
To make the algorithm practical it is necessary to reduce the number of committee members that are generated. Preferably this should be done adaptively so that accuracy on future data is not affected negatively. The first design decision concerns which pruning operations to apply. The second problem is how to decide whether pruning should occur. Boosting is a sequential process where new models are built based on data weighted according to the predictions of previous models. Hence it may be detrimental to prune models somewhere in the middle of a committee because subsequent models have been generated by taking the predictions of previous models into account. Consequently the only model that we consider for pruning is the last model in the sequence. This makes pruning a straightforward (and computationally very efficient) procedure: the existing committee is compared to the new committee that has an additional member based on the latest chunk of data. If the former is judged more accurate, the last model is discarded and the boosting process continues with the next chunk of data. Hence the pruning process makes it possible to skip chunks of data that do not contribute positively to the committee’s accuracy. As the experimental results presented in Section 3 show, this is especially useful when small chunks are used to build the committee. The experimental results also show that it is not advisable to stop building the committee when a “bad” chunk of data is encountered because later chunks of data may prove useful and lead to models that improve the committee’s accuracy. The second aspect to pruning is the choice of evaluation criterion. The pruned model needs to be compared to the unpruned one. Pruning should occur if it does not negatively affect the committee’s generalization performance. Fortunately our target application domains share a common property: they exhibit an abundance of data. This means we can be generous and reserve some of the data for pruning. We call this data “validation data.” This data is held completely separate from the data used for training the models. In our implementation the first N instances encountered (where N is the size of the validation dataset) are skipped by the boosting process. Consequently the first chunk of data that generates a potential committee member starts with instance N + 1.
Racing Committees for Large Datasets
157
START with an empty committee K0 AND validation data V REPEAT FOR next data chunk Ci DO BEGIN IF (i > 1) THEN weight chunk Ci according to the predictions of K0..i−1 learn model Mi for chunk Ci IF (loglikelihood for K0..i−1 + Mi on V > loglikelihood for K0..i−1 on V ) THEN add Mi to K0..i−1 END UNTIL no more chunks Fig. 2. Incremental boosting with pruning.
Accuracy on the validation data is the obvious performance measure. However, we found empirically that this is not a good pruning criterion. Preliminary results showed that it results in useful models being skipped because they do not change the accuracy immediately although they do improve accuracy in conjunction with models that are built later in the process. Logistic regression attempts to maximize the likelihood of the data given the model. An alternative candidate for measuring performance is therefore the loglikelihood on the validation data. This measures the accuracy of the class probability estimates that are generated by the committee. It turns out that using the loglikelihood avoids overpruning because it is more sensitive to whether a potential committee member manages to extract useful additional information. The resulting pruning algorithm based on the loglikelihood is depicted in Figure 2. Pruning reduces the size of the committee according to the properties of the data. Ideally no further models are added to the committee when the information in the data is exhausted. If this is the case there exists an upper bound on the number of models that are generated and the time complexity becomes linear in the number of training instances, allowing very large datasets to be processed effectively. Of course, apart from affecting running time, pruning also reduces the amount of memory that is needed to store the committee. 2.3
Racing Committees
Experimental results show that the performance of the committee varies, sometimes dramatically, with the chunk size. The chunk size should be large enough for each individual committee member to become a reliable predictor. However, as the chunk size increases, returns for each individual committee member diminish. At some point it becomes more productive to increase the diversity of the committee by starting with a new chunk. The best chunk size depends on the properties of the particular dataset and the weak learner used in the boosting process.
158
E. Frank et al.
Given these observations it appears impossible to determine an appropriate chunk size a priori. Consequently the only sensible strategy is to decide on a range of chunk sizes and to run the different committees corresponding to these different chunk sizes in parallel—i.e. to “race” them off against each other. Then we can keep track of which committee performs best and use the best-performing committee for prediction. Typically the best-performing chunk size changes as more data becomes available. The validation data, which is also used for pruning, can be used to compare the performance of the committees. However, in contrast to pruning, where the loglikelihood is employed to measure performance, here it is more appropriate to use percent correct because we want to use the committee that maximizes percent correct for future data.1 The question remains as to how many committees to run in parallel and which set of chunk sizes to use. Ultimately this depends on the computing resources available. If the number of committees is constant then the time and space complexity of racing them is the same as the corresponding complexities for its “worst-case” member. Consequently, assuming that pruning works and after a certain number of iterations no further models are added to the committee, the overall time-complexity is linear in the number of instances, and the space complexity is constant. In our experiments we used the following five chunk sizes: 500, 1,000, 2,000, 4,000, and 8,000. We kept the maximum chunk size relatively small because decision stumps are particularly weak classifiers and the returns on adding more data diminish quickly. Doubling the chunk size from one candidate to the next has the advantage that whenever the committee corresponding to the largest chunk size may have changed this is also true for all the smaller ones, and a comparison on the validation data at this point is fair because all the committees have “seen” the same amount of training data.
3
Experimental Results
To evaluate the performance of racing unpruned and pruned committees we performed experiments on six datasets ranging in size from approximately 30,000 to roughly 500,000 instances. The properties of these datasets are shown in Table 1.2 We obtained them from the UCI repositories [1,7]. The kdd cup ’99 data is a reduced version of the full dataset (reduced so that incremental boosting without pruning could be applied to this data). The “Train” column shows the amount of data that is used for training the committee (excluding the validation data). We attempted to set a sufficient amount of data aside for validation and testing to obtain accurate performance estimates. In our experiments the validation set size was set to half the size of the test set. Before we split the data 1 2
Of course, if the application domain requires accurate probability estimates, it is more appropriate to use the loglikelihood for choosing the best committee. The first three datasets are rather small but we chose to include them in our comparison because of the lack of publicly available large datasets.
Racing Committees for Large Datasets
159
Table 1. Datasets and their characteristics Dataset anonymous adult shuttle census income kdd cup ’99 covertype
Train Validation Test 30211 2500 5000 33842 5000 10000 43000 5000 10000 224285 25000 50000 475000 25000 50000 506012 25000 50000
Numeric Nominal Classes 0 293 2 6 8 2 9 0 7 8 33 2 34 7 22 10 44 7
into training, validation, and test data we randomized it to obtain independent and identically distributed samples. The first row of Figure 3 shows the results for the anonymous data. The leftmost graph shows percent incorrect on the test set for the unpruned committees as an increasing amount of training data is processed. Points mark the graph corresponding to the committee that performs best on the validation data (i.e. the committee that would be used for prediction at that point under the racing scheme). The middle graph shows the same for the pruned committees, and the rightmost graph shows the committee sizes for the pruned committees.3 The worst-performing chunk size is 8,000 because there is insufficient data to build a large enough committee. The final ranking of committees is the same with and without pruning (and the resulting error rates are comparable). Pruning appears to smooth fluctuations in error on the test data. Pruning also substantially reduces the committee size for small chunk sizes. After 30,000 training instances, 60 models are built without pruning for chunk size 500, with pruning there are only 16 (and it appears that the number of models has reached a plateau). No pruning is done for chunk sizes 4,000 and 8,000. It appears as if pruning should have occurred for chunk size 8,000. However, the loglikelihood on the validation data does increase after models are added and consequently no pruning occurs. The second row of Figure 3 shows the results for the adult data. Substantial pruning occurs for chunk sizes 500 and 1,000. In both cases it smoothes fluctuations in the performance and results in improved final error. It is interesting to see that the final size of the pruned committee for chunk size 1,000 is larger than the size of the committee for chunk size 500. Pruning appears to behave correctly because the final error is lower for the former. The shuttle data in the third row of Figure 3 is different from the previous two datasets in that very high accuracy scores can be achieved. The results show that pruning produces substantially smaller committees for chunk sizes 500, 1,000, and 2,000. However, for chunk sizes 500 and 1,000 it also results in fractionally lower accuracy. Choosing the best-performing committee on the validation data under the racing scheme results in approximately the same final error rate both with and without pruning. 3
The size of the unpruned committees is not shown because it increases linearly with the amount of training data.
160
E.Frank et a1
Racing Committees for Large Datasets
161
162
E. Frank et al.
Table 2. Percent incorrect for standard LogitBoost compared to our racing scheme LogitBoost #Iterations Racing w/o pruning Racing w pruning Dataset anonymous 27.00% 60 28.24% 27.56% adult 13.51% 67 14.58% 14.72% shuttle 0.01% 86 0.08% 0.07% census-income 4.43% 448 4.90% 4.93%
The next dataset we consider is census-income. The first row of Figure 4 shows the results. The most striking aspect is the effect of pruning with small chunk sizes. In this domain the fluctuation in error is extreme without pruning. With pruning this erratic behavior disappears and error rates decrease dramatically. Pruning also results in a marginally lower final error rate for the largest chunk sizes. The results also show that the size of the pruned committees starts to level out after a certain amount of training data has been seen. Note that even though chunk size 8,000 results in the most accurate pruned committee on the test data, chunk size 4,000 is chosen for prediction based on superior performance on the validation set. The kdd cup ’99 domain is similar to the shuttle domain in that very high accuracy can be achieved. As in the shuttle domain overpruning occurs for small chunk sizes (however, note that the degradation in performance corresponds to fractions of a percent). Although this is difficult to see on the graphs, pruning marginally improves performance for the largest chunk size (8,000). Under the racing scheme the final performance is approximately the same both with and without pruning. Because the dataset is so large, pruning results in substantial savings in both memory and runtime. The behavior on the largest dataset (third row of Figure 4) is similar to that seen on the census-income dataset. The only difference is that the pruned version chooses chunk size 4,000 on census-income, whereas 8,000 is chosen for covertype. Pruning substantially increases accuracy for chunk sizes 500, 1,000, and 2,000 eliminating the erratic behavior of the unpruned committees. The best-performing committee (both pruned and unpruned) is based on a chunk size of 8,000. Pruning does not improve the accuracy of the final predictor under the racing scheme. However, it does lead to substantial savings in both memory and runtime. The final committee for chunk size 8,000 is less than half the size of the unpruned version. Table 2 compares the final error under the racing scheme to standard LogitBoost (i.e. where the weak learner is applied to the full training set in each boosting iteration). We set the number of iterations for standard LogitBoost to be the same as the number of committee members in the largest unpruned committee (i.e. the one built from chunk size 500). The table does not include results for the two largest datasets because processing them with standard LogitBoost was beyond our computing resources. As might be expected, standard LogitBoost is slightly more accurate on all four test sets. However, the results are very close.
Racing Committees for Large Datasets
4
163
Related Work
Breiman [3] appears to have been the first to apply boosting (or “arcing” [2]) to the problem of processing large datasets by using a different subsample in each iteration of the boosting algorithm. He shows that this produces more accurate predictions than using bagging in the same fashion. He also shows that incremental boosting (if used in conjunction with an appropriate subsample size) produces classifiers that are about as accurate as the ones generated by standard boosting applied to the full dataset. However, his work does not address the problem of how to decide which committee members to discard. Fan et al. [4] propose an incremental version of AdaBoost that works in a similar fashion. Their method retains a fixed-size “window” of weak classifiers that contains the k most recently built classifiers. This makes the method applicable to large datasets in terms of memory and time requirements. However, it remains unclear how an appropriate value for k can be determined. Street and Kim [14] propose a variant of bagging for incremental learning based on data chunks that maintains a fixed-size committee. In each iteration it attempts to identify a committee member that should be replaced by the model built from the most recent chunk of data. Because the algorithm is based on bagging (i.e. all data points receive equal weight and a simple majority vote is performed to make a prediction), the algorithm has limited potential to boost the performance of the underlying weak classifiers. Oza and Russell [10] propose incremental versions of bagging and boosting that differ from our work because they require the underlying weak learner to be incremental. The method is of limited use for large datasets if the underlying incremental learning algorithm does not scale linearly in the number of training instances. Unfortunately, the time complexity of most incremental learning algorithms is worse than linear. Prodromidis and Stolfo [11,12] present pruning methods for ensemble classifiers built from different (unweighted) subsets of a dataset. These methods require an unpruned ensemble to be built first before pruning can be applied. The pruned ensemble cannot be updated incrementally as new data arrives. Similarly, Margineantu and Dietterich [9] investigate pruning methods for ensembles built by the standard AdaBoost algorithm (i.e. where a weak classifier is built from the entire dataset in each boosting iteration). Again, their method is applied once an unpruned ensemble has been generated.
5
Conclusions
This paper has presented a method for efficiently processing large datasets using standard learning techniques by wrapping them into an incremental boosting algorithm. The main contribution of this paper is a pruning method for making the procedure efficient in terms of memory and runtime requirements. The accuracy of the resulting committee depends on an appropriately chosen chunk
164
E. Frank et al.
size. Experimental results obtained by racing candidate solutions based on different chunk sizes demonstrate the effectiveness of our method on six real-world datasets. Although our technique can be used in an online setting, it cannot be applied in domains with concept drift (i.e. where the target concept changes over time) because it assumes that all the incoming data is independent and identically distributed. Acknowledgments. We would like to thank Bernhard Pfahringer for his valuable comments.
References 1. C. Blake, E. Keogh, and C.J. Merz. UCI repository of machine learning databases, 1998. [www.ics.uci.edu/∼mlearn/MLRepository.html]. 2. Leo Breiman. Arcing classifiers. The Annals of Statistics, 26(3):801–849, 1998. 3. Leo Breiman. Pasting small votes for classification in large databases and on-line. Machine Learning, pages 85–103, 1999. 4. Wei Fan, Salvatore J. Stolfo, and Junxin Zhang. The application of AdaBoost for distributed, scalable and on-line learning. In 5th ACM SIGKDD Int. Conf. on Knowledge Discovery and Data Mining, pages 362–366, 1999. 5. Yoav Freund and Robert E. Schapire. Experiments with a new boosting algorithm. In Proc. 13th Int. Conf. on Machine Learning, pages 148–156. Morgan Kaufmann, 1996. 6. Jerome Friedman, Trevor Hastie, and Robert Tibshirani. Additive logistic regression: a statistical view of boosting. The Annals of Statistic, 38(2):337–374, 2000. 7. S. Hettich and S. D. Bay. The UCI KDD archive, 1999. [http://kdd.ics.uci.edu]. 8. G. H. John and P. Langley. Static versus dynamic sampling for data mining. In 2nd ACM SIGKDD Int. Conf. on Knowledge Discovery in Databases and Data Mining, pages 367–370, 1996. 9. D. D. Margineantu and T. G. Dietterich. Pruning adaptive boosting. In Proc. of the 14th Int. Conf. on Machine Learning, pages 211–218, 1997. 10. Nikunj Oza and Stuart Russell. Experimental comparisons of online and batch versions of bagging and boosting. In 7th ACM SIGKDD Int. Conf. on Knowledge Discovery in Databases and Data Mining, pages 359–364, 2001. 11. A. L. Prodromidis, S. J. Stolfo, and P. K. Chan. Pruning classifiers in a distributed meta-learning system. In Proc. of 1st National Conference on New Information Technologies, pages 151–160, 1998. 12. Andreas L. Prodromidis and Salvatore J. Stolfo. Cost complexity-based pruning of ensemble classifiers. Knowledge and Information Systems, 3(4):449–469, 2001. 13. A. Srinivasan. A study of two sampling methods for analysing large datasets with ILP. Data Mining and Knowledge Discovery, 3(1):95–123, 1999. 14. W. Nick Street and YongSeog Kim. A streaming ensemble algorithm (SEA) for large-scale classification. In 7th ACM SIGKDD Int. Conf. on Knowledge Discovery in Databases and Data Mining, pages 377–382, 2001.
From Ensemble Methods to Comprehensible Models C. Ferri, J. Hern´ andez-Orallo, and M.J. Ram´ırez-Quintana DSIC, UPV, Camino de Vera s/n, 46020 Valencia, Spain. {cferri,jorallo,mramirez}@dsic.upv.es
Abstract. Ensemble methods improve accuracy by combining the predictions of a set of different hypotheses. However, there are two important shortcomings associated with ensemble methods. Huge amounts of memory are required to store a set of multiple hypotheses and, more importantly, comprehensibility of a single hypothesis is lost. In this work, we devise a new method to extract one single solution from a hypothesis ensemble without using extra data, based on two main ideas: the selected solution must be similar, semantically, to the combined solution, and this similarity is evaluated through the use of a random dataset. We have implemented the method using shared ensembles, because it allows for an exponential number of potential base hypotheses. We include several experiments showing that the new method selects a single hypothesis with an accuracy which is reasonably close to the combined hypothesis. Keywords: Ensemble Methods, Decision Trees, Comprehensibility in Machine Learning, Classifier Similarity, Randomisation.
1
Introduction
Comprehensibility has been the major advantage that has been advocated for supporting some machine learning methods such as decision tree learning, rule learners or ILP. One major feature of discovery is that it gives insight from the models, properties and theories that can be obtained. A model that is not comprehensible may be useful to obtain good predictions, but it cannot provide knowledge about how predictions are made. With the goal of improving model accuracy, there has been an increasing interest in constructing ensemble methods that combine several hypotheses [4]. The effectiveness of combination is further increased the more diverse and numerous the set of hypotheses is [10]. Decision tree learning (either propositional or relational) is especially benefited by ensemble methods [18,19]. Well-known techniques for generating and combining hypotheses are boosting [9,18], bagging [1,18], randomisation [5], stacking [22] and windowing [17].
This work has been partially supported by CICYT under grant TIC2001-2705-C0301, Generalitat Valenciana under grant GV00-092-14 and Acci´ on Integrada HispanoAlemana HA2001-0059.
S. Lange, K. Satoh, and C.H. Smith (Eds.): DS 2002, LNCS 2534, pp. 165–177, 2002. c Springer-Verlag Berlin Heidelberg 2002
166
C. Ferri, J. Hern´ andez-Orallo, and M.J. Ram´ırez-Quintana
Although ensemble methods significantly increase accuracy, they have some drawbacks, mainly the loss of comprehensibility of the model and the large amount of memory required to store the hypotheses [13]. Recent proposals have shown that memory requirements can be considerably reduced (in [16], a method called miniboosting reduces the ensemble to just three hypotheses, with 40% less of the improvement that would be obtained by a 10trial AdaBoost). Nonetheless, the comprehensibility of the resulting combined hypothesis is not improved. A combined hypothesis is usually a voting of many hypotheses and it is usually treated as a black box, giving no insight at all. However, one major goal of the methods used in discovery science is comprehensibility. The question is how to reduce to one single hypothesis from the combination of m hypotheses without losing too much accuracy with respect to the combined hypothesis. Instead of using classical methods for selecting one hypothesis, such as the hypothesis with the lowest expected error, or the one with the smallest size (Occam’s razor), we will select the single hypothesis that is most similar to the combined hypothesis. This single hypothesis will be called an archetype or representative of the ensemble and can be seen as an ‘explanation’ of the ensemble. To do this, the main idea is to consider the combination as an oracle that would allow us to measure the similarity of each single hypothesis with respect to this oracle. More precisely, for a hypothesis or solution h and an unlabelled example e, let us define h(e) as the class or label assigned to e by h. Consider an ensemble of solutions E = h1 , h2 , · · · hm and a method of combination χ. By Σχ,E we denote the combined solution formed by using the method χ on E. Thus, Σχ,E (e) is the class assigned to e by the combined solution. Now, we can use Σχ,E as an oracle, which, generally, gives better results than any single hypothesis [4]. The question is to select a single hypothesis hi from E such that hi is the most similar (semantically) to the oracle Σχ,E . This rationale is easy to understand following the representation used in a statistical justification for the construction of good ensembles presented by Dietterich in [4]. A learning algorithm is employed to find different hypotheses {h1 , h2 , · · · , hm } in the hypothesis space or language H. By constructing an ensemble out of all these classifiers, the algorithm can “average” their votes and reduce the risk of choosing a wrong classifier. Figure 1 depicts this situation. The outer curve denotes the hypothesis space H. The inner curve denotes the set of hypotheses that give a reasonably good accuracy on the training data and hence could be generated by the algorithm. The point labelled by F is the true hypothesis. H h1 h2 F
hc
h3
h4 h5
Fig. 1. Representation of an ensemble of hypotheses
From Ensemble Methods to Comprehensible Models
167
If an ensemble hc is constructed by combining the accurate hypotheses, hc is a good approximation to F . However, hc is an ensemble, which means that it needs to store {h1 , h2 , · · · , h5 } and it is not comprehensible. For this reason, we are interested in selecting the single hypothesis from {h1 , h2 , · · · , hm } that would be closest to the combination hc . Following the previous rationale, this single hypothesis would be close to F . In the situation described in Figure 1, we would select h4 as the archetype or representative of the ensemble. A final question, also pointed out by [4], is that a statistical problem arises when the amount of training data available is too small compared to the size of the hypothesis space H. The selection of a good archetype would not be possible if a sufficient amount of data is not available for comparing the hypotheses. The reserve of part of the training data is generally not a good option because it would yield a smaller training dataset and the ensemble would have a lower quality. This problem has a peculiar but simple solution: the generation of random unlabelled datasets. Although the technique presented in this work is applicable to many kinds of ensemble methods, we will illustrate it with shared ensembles, because the number of hypotheses, in this kind of structure, grows exponentially wrt. the number of iterations. Therefore, there is a much bigger population which the representative can be extracted from. The paper is organised as follows. First, in section 2, we discuss the use of a similarity measure and we adapt several similarity metrics we will use. Section 3 explains how artificial datasets can be employed to estimate the similarity between every classifier and their combination. Section 4 presents the notion of shared ensemble, its advantages for our goals and how it can be adapted for the selection of the most similar hypothesis with respect to the combination. A thorough experimental evaluation is included in Section 5. Finally, the last section presents the conclusions and proposes some future work.
2
Hypothesis Similarity Metrics
As we have stated, our proposal is to select the single hypothesis which is most similar to the combined one. Consequently, we have to introduce different measures of hypothesis similarity. These metrics and an additional dataset will allow the estimation of a value of similarity between two hypotheses. In the following, we will restrict our discussion to classification problems. Several measures of hypothesis similarity (or diversity) have been considered in the literature with the aim of obtaining an ensemble with high diversity [12]. However, some of these are defined for a set of hypotheses and others for a pair of hypotheses. We are interested in these “pairwise diversity measures”, since we want to compare a single hypothesis with an oracle. However, not all of these measures can be applied here. First, the approach presented by [12] requires the correct class to be known. The additional dataset should be labelled, which means that part of the training set should be reserved for the estimation of similarities. Secondly, some other metrics are only applicable to two classes. As
168
C. Ferri, J. Hern´ andez-Orallo, and M.J. Ram´ırez-Quintana
a result, in what follows, we describe the pairwise metrics that can be estimated by using an unlabelled dataset and that can be used for problems with more than two classes. Given two classifiers ha and hb , and an unlabelled dataset with n examples with C classes, we can construct a C × C contingency or confusion matrix Mi,j that contains the number of examples e such that ha (e) = i and hb (e) = j. With this matrix, we can define the following similarity metrics: – θ measure: It is just based on the idea of determining the probability of both classifiers agreeing: θ=
C Mi,i n i=1
Its value is between 0 and 1. An inverse measure, known as discrepancy is also considered by [12]. – κ measure: The previous metric has the problem that when one class is much more common than the others or there are only two classes, this measure is highly affected by the fact that some predictions may match just by chance. Following [13], we define the Kappa measure, which was originally introduced as the Kappa statistic (κ) [3]. This is just a proper normalisation based on the probability that two classifiers agree by chance: θ2 =
C C C Mi,j Mj,i ( · ) n n i=1 j=1 j=1
As a result, the Kappa statistic is defined as: κ=
θ − θ2 1 − θ2
Its value is usually between 0 and 1, although a value lower than 0 is possible, meaning that the two classifiers agree less than two random classifiers agree. – Q measure: The Q measure is defined as follows [12]: C
i=1
Mi,i −
i=1
Mi,i +
Q = C
C
i=1,j=1,i=j
Mi,j
i=1,j=1,i=j
Mi,j
C
This value varies between -1 and 1. Note that this measure may have problems if any component of M is 0. Thus it is convenient to apply smoothing in M to compute the measure. We will add 1 to every cell. Obviously, the greater the reference dataset is, all of the previous metrics give a better estimate of similarity. In our case, and since the previous measures use the contingency matrix, we can have huge reference datasets available: random invented datasets.
3
Random Invented Datasets
In many situations, a single hypothesis may be the one which is the most similar to the combined hypothesis with respect to the training set, however it may not
From Ensemble Methods to Comprehensible Models
169
be the most similar one in general (with respect to other datasets). In some cases, e.g. if we do not use pruning, then all the hypotheses (and hence the combined solution) may have 100% accuracy with respect to the training set, and all the hypotheses are equally “good”. Therefore, it is suitable or even necessary to evaluate similarity with respect to an external (and desirably large) reference dataset. In many cases, however, we cannot reserve part of the training set for this, or it could be counterproductive. The idea then is to use the entire training set to construct the hypotheses and to use a random dataset to select one of them. In this work, we consider that the examples in the training set are equations of the form f (· · ·) = c, where f is a function symbol and c is the class of the term f (· · ·). Given a function f with a arguments, an unlabelled random example is any instance of the term f (X1 , X2 , · · · , Xa ), i.e., any term of the form f (v1 , v2 , · · · , va ) obtained by replacing every attribute Xi by values vi from the attribute domain (attribute type). Note that an unlabelled random example is not an equation (a full example) because we include no information about the correct class. We will use the following technique to generate each random unlabelled example: each attribute Xi of a new example is obtained as the value vi in a different example f (v1 , . . . , vi , . . . , va ) selected from the training set by using a uniform distribution. This procedure of generating instances assumes that all the attributes are independent, and just maintains the probabilities of appearance of the different values observed in each attribute of the training dataset.
4
Shared Ensembles
A multi-tree is a data structure that permits the learning of ensembles of trees that share part of their branches. These are called “shared ensembles”. In the particular case of trees, a multi-tree can be based on an AND/OR organisation, where some alternative splits are also explored. Note that a multi-tree is not a forest [10], because a multi-tree shares the common parts of different trees, whereas a forest is just a collection of trees. In a previous work [6], we presented an algorithm for the induction of multitrees which is able to obtain several hypotheses, either by looking for the best one or by combining them in order to improve accuracy. To do this, once a node has been selected to be split (an AND-node) the possible splits below (OR-nodes) are evaluated. The best one, according to the splitting criterion is selected and the rest are suspended and stored. After is completed the first solution, when a new solution is required, one of the suspended nodes is chosen and ‘woken’, and the tree construction follows under this node. This way, the search space is an AND/OR tree [14] which is traversed, thus producing an increasing number of solutions as the execution time increases. In [7], we presented several methods for growing the multi-tree structure. Since each new solution is built by completing a different alternative OR-node branch, our method differs from other approaches such as the boosting or bagging methods [1,9,18] which would induce a new decision tree for each solution.
170
C. Ferri, J. Hern´ andez-Orallo, and M.J. Ram´ırez-Quintana
Note that in a multi-tree structure there is an exponential number of possible hypotheses with respect to the number of alternative OR-nodes explored. Consequently, although the use of multi-trees for combining hypotheses is more complex, it is more powerful because it allows us to combine many more hypotheses using the same resources. Other previous works have explored the entire the search space of the AND/OR tree to make the combination [2], inspired by Context Tree Weighting (CTW) models [20], whereas we only explore a subset of the best trees. 4.1
Shared Ensemble Combination
Given several classifiers that assign a probability to each prediction (also known as soft classifiers) there are several combination methods or fusion strategies that can be applied. Let us denote by pk (cj |x) an estimate of the posterior probability that classifier k assigns class cj for example x. If we consider all the estimates equally reliable we can define several fusion strategies: majority vote, sum or arithmetic mean, product or geometric mean, maximum, minimum and median. Some works have studied which strategy is best. In particular, [11] concludes that, for two-class problems, minimum and maximum are the best strategies, followed by average (arithmetic mean). In decision tree learning, the pk (cj |x) depend on the leaf node where each x falls. More precisely, these probabilities depend on the proportion of training examples of each class that have fallen into each node during training. The reliability of each node usually depends on the cardinality of the node. Let us define a class vector vk,j (x) as the vector of training cases that fall in each node k for each class j. For leaf nodes the values would be the training cases of each class that have fallen into the leaf. To propagate upwards these vectors to internal nodes, we must clarify how to propagate through AND and OR nodes. This is done for each new unlabelled example we want to make a prediction for. For the AND-nodes, the answer is clear: an example can only fall through an AND-node. Hence, the vector would be the one of the child where the example falls. OR-nodes, however, must do a fusion whenever different alternative vectors occur. This is an important difference in shared ensembles: fusion points are distributed all over the multi-tree structure. We have implemented several fusion strategies. Nonetheless, it is not the goal of this paper to evaluate different methods for combining hypotheses but to select a single hypothesis. Thus, for the sake of simplicity, in this paper we will only use the maximum strategy because it obtains the best performance, according to our own experiments and those of [11]. 4.2
Selecting an Archetype from a Shared Ensemble
In a shared ensemble, we are not interested (because it would be unfeasible) to compute the similarity of each hypothesis with respect to the combined hypothesis, because there would be an exponential number of comparisons. What we are interested in is a measure of similarity for each node with respect to the
From Ensemble Methods to Comprehensible Models
171
combined solution, taking into account only the examples of the invented dataset that fall into a node. The general idea is, that once the multi-tree is constructed, we use its combination to predict the classes for the previously unlabelled invented dataset. Given an example e from the unlabelled invented dataset, this example will fall into different OR-nodes and finally into different leaves, giving different class vectors. Then, the invented dataset is labelled by voting these predictions in the way explained in the previous subsection. After this step, we can calculate a contingency matrix for each node, in the following way. For each node (internal or leaf), we have a C × C contingency matrix called M , initialised to 0, where C is the number of classes. For each example in the labelled invented dataset, we increment the cell Ma,b of each leaf where the example falls by 1, with a being the predicted class by the leaf and b being the predicted class by the combination. When all the examples have been evaluated and the matrices in the leaf nodes have been assigned, then we propagate the matrices upwards as follows: – For the contingency matrix M of AND-nodes we accumulate the contingency matrix of their m children nodes: (M1 + M2 + · · · + Mm ). – For the contingency matrix M of OR-nodes, the node of their children with greater Kappa (or other similarity measure) is selected and its matrix is propagated upwards. The selected node is marked. This ultimately generates the hypothesis that is most similar to the combined hypothesis, using a particular invented dataset and a given similarity measure. K=0.5
K=0.52
24
7
19
5
5
14
7
19
X>6
X<6
Y=a
4
13
1
3
9
1
15
K=0.52
Y=a
Y=b
14
14
4
3
9
K=0.87 10
3
6
4
2
5
6
4
Y=b
X>3
K=−0.18 13
1
10
7
1
15
10
3
X<3
X>9
X<9
10
3
4
1
7
0
6
1
2
4
8
3
2
5
1
4
1
7
0
8
2
2
8
1
Fig. 2. Selection of a single decision tree from the multi-tree structure.
Figure 2 shows the selection of one hypothesis from a multi-tree according to the contingency matrix. The AND-nodes are represented with an arc. The leaves are represented by rectangles. First, we fill the matrices of the leaves. Then, we propagate these upwards as has been detailed previously. Finally, when we reach the top of the tree, it is straightforward to extract the solution by simply descending the multi-tree by the marked nodes. In the figure, the marked nodes
172
C. Ferri, J. Hern´ andez-Orallo, and M.J. Ram´ırez-Quintana
are represented by the dashed lines, and the leaves of the selected hypothesis are shadowed. Therefore, we can summarise the approach in five different steps: 1. Multi-tree generation: The first step consists in the generation of a multitree from a training dataset. There are some criteria which affect the quality of the multi-tree: the splitting criterion, the pruning method, and the criterion for the selection of the suspended node to be woken. 2. Invented dataset: In this phase, an unlabelled invented dataset is created, by a random dataset. 3. Multi-tree combination: The invented dataset is labelled by the combination of the shared ensemble. A method of combination of hypotheses can be specified. 4. Calculation and propagation of contingency matrices: A contingency matrix is assigned to each node of the multi-tree, using the labelled invented dataset and a similarity metric. 5. Selection of a solution: An archetype hypothesis is extracted from the multi-tree by descending the multi-tree through the marked nodes.
5
Experiments
In this section, we present an experimental evaluation of our approach, as it is implemented in the SMILES system [8]. SMILES is a multi-purpose machine learning system which includes (among many other features) the implementation of a multi-tree learner. For the experiments, we used GainRatio [17] as splitting criterion. We chose a random method [7] for populating the shared ensemble (after a solution is found, a suspended OR-node is woken at random) and we used the maximum strategy for combination. We used several datasets from the UCI dataset repository [15]. Table 1 shows the dataset name, the size in number of examples, the number of classes, the nominal and numerical attributes. Table 1. Information about datasets used in the experiments. # Dataset Size Classes Nom.Attr. Num.Attr. 1 monks1 566 2 6 0 2 monks2 601 2 6 0 3 monks3 554 2 6 0 tic-tac 958 2 8 0 4 5 house-votes 435 2 16 0 3 7 1 6 post-operative 87 7 balance-scale 625 3 0 4 8 soybean-small 35 4 35 0 9 dermatology 358 6 33 1 10 cars 1728 4 5 0 11 tae 151 3 2 3 12 new-thyroid 215 3 0 5 13 ecoli 336 8 0 7
From Ensemble Methods to Comprehensible Models
173
Since there are many sources of randomness, we have performed the experiments by averaging 10 results of a 10-fold cross-validation. This makes a total of 100 runs (each one with a different multi-tree construction, random dataset and hypothesis selection process) for each pair of method and dataset. In the experiments, we will use the following notation: – First Solution: this is the solution given by just one hypothesis (the first hypothesis that is obtained). This is similar to C4.5 [17]. – Combined Solution: this is the solution given by combining the results of the ensemble (in our case, the multi-tree, as described in the previous section). – Archetype Solution: this is the single solution which is most similar to the combined solution. – Occam Solution: this is the single solution with the lowest number of rules, i.e., the shortest solution. It is not our purpose to evaluate the improvement of the Combined Solution over the First Solution using shared ensembles. We have done that in previous works [7]. We have not included the results using post-pruning because it does not improve the performance of any of the four kinds of solutions. Our goal is to show that a significant gain can be obtained from the First Solution to the Archetype and Occam methods as long as the size of the ensemble increases. Another question to be answered is to determine which method to extract a single solution from an ensemble is better: Archetype or Occam. 5.1
Evaluating Similarity Metrics
Table 2 shows the accuracy for each pair composed of a dataset and a method and the geometric means for each method. The methods studied are First, Combined and Archetype. The latter uses three different similarity metrics κ, θ and Q. The multi-tree has been generated exploring 100 suspended OR-nodes. Table 2. Comparison between measures of similarity. # 1 2 3 4 5 6 7 8 9 10 11 12 13 gmeans
1st 92.3 74.8 97.5 78.2 93.6 60.9 76.8 97.3 89.8 89.0 62.9 92.6 77.5 82.41
Comb 100 77.4 97.5 82.7 96.0 66.3 83.1 96.5 93.6 91.0 64.5 92.6 79.9 85.45
Arc. κ 100 76.1 97.6 78.2 94.4 63.8 80.1 96.5 90.6 89.6 61.9 92.8 79.4 83.78
Arc. θ Arc. Q 100 100 76.2 75.8 97.6 97.6 78.3 78.5 93.9 94.2 64.3 61.9 80.1 79.8 91.0 47.0 89.9 74.3 89.6 89.3 62.9 49.8 92.9 91.4 78.9 76.7 83.45 76.24
As expected, hypothesis combination improves the accuracy w.r.t. the first single tree. The use of the Archetype method also obtains good results. On the
174
C. Ferri, J. Hern´ andez-Orallo, and M.J. Ram´ırez-Quintana Table 3. Influence of the size of the invented dataset. 10 100 1000 10000 100000 # Comb Arc Arc Arc Arc Arc 1 99.8 72.3 93.3 99.8 100 99.9 2 77.3 64.6 61.0 75.2 76.1 76.2 97.6 82.9 94.5 97.6 97.6 97.6 3 82.9 65.9 70.3 78.0 78.2 78.6 4 5 95.8 73.7 92.4 94.4 94.4 93.8 63.5 6 67.5 69.1 63.6 63.9 63.8 79.9 7 83.0 62.5 75.4 79.4 80.1 96.5 8 95.0 68.8 93.3 95.0 96.5 9 93.6 45.6 84.7 90.5 90.6 89.9 10 91.0 71.0 75.4 88.1 89.6 89.8 11 63.7 44.3 54.3 59.1 61.9 61.2 12 92.5 73.8 89.3 91.3 92.8 92.6 13 80.0 46.8 73.9 77.9 79.4 79.0 gmeans 85.36 63.57 77.40 82.88 83.78 83.57
other hand, the results show that the Archetype method is very dependent on the measure of similarity used: κ seems to be the best metric and Q the worst (it even obtains lower accuracy than the first single hypothesis). 5.2
Influence of the Size of the Invented Dataset
Similarity is approximated through the use of an invented dataset. Let us study the influence of its size, varying from 10 to 100,000 examples. The similarity metric and the size of the multi-tree are fixed to κ and 100 alternative opened OR-trees, respectively. Table 3 shows that in order to obtain a good archetype hypothesis, the similarity metric has to be computed as accurately as possible. Although it depends on the dataset, a size of 10,000 invented examples seems to be sufficient. 5.3
Influence of the Size of the Ensemble
The effect of the size of the multi-tree is evaluated in Table 4. In this table, we show the accuracy of the first single solution and the accuracy of the combination, the archetype solution and the Occam solution for multi-trees created by exploring 10, 100, and 10001 alternative OR-nodes. We also include the geometric average number of solutions in the multi-tree (#Sol). Note that with 100 OR-nodes, we obtain millions of solutions with much less required memory than 100 non-shared hypotheses. The results are quite encouraging: by simply exploring 10 OR-nodes, the archetype solution surpasses the first solution and the Occam solution. This difference is increased as long as the multi-tree is populated. This is mainly due to the improvement in the accuracy of the combined solution and the fact that the archetype hypothesis can actually get close to it. The Occam solution 1
The experiments for datasets 9 and 13 have been performed exploring only 300 and 500 alternative OR-nodes, respectively.
From Ensemble Methods to Comprehensible Models
175
Table 4. Influence of the size of the multi-tree. # 1 2 3 4 5 6 7 8 9 10 11 12 13 gm.
1 10 100 1st Comb Arc Occ #Sol Comb Arc Occ 92.3 96.1 96.0 96.5 107 100 100 100 74.8 74.9 74.3 74.3 148 77.4 76.1 72.5 97.5 97.7 97.7 97.6 46 97.5 97.6 97.5 78.2 79.0 78.1 78.3 257 82.7 78.2 78.6 93.6 94.9 94.2 93.9 63 96.0 94.4 93.6 66.3 63.8 62.3 60.9 63.8 61.8 60.0 55 76.8 77.9 77.2 76.8 131 83.1 80.1 76.7 97.3 97.0 98.0 97.5 23 96.5 96.5 96.8 93.6 90.6 90.2 89.8 91.3 90.6 90.1 92 89.0 89.6 89.1 89.0 151 91.0 89.6 89.1 62.9 62.5 62.3 61.9 97 64.5 61.9 62.1 92.6 92.8 93.0 92.6 93.2 92.6 92.6 26 77.5 79.1 77.6 77.8 57 79.9 79.4 78.4 82.41 83.49 82.85 82.55 78.31 85.45 83.78 82.91
1000 #Sol Comb Arc Occ 8.7 × 108 100 100 100 2.6 × 1010 82.3 82.1 70.4 80 × 104 97.7 97.7 97.6 2.7 × 1012 84.6 79.8 79.5 5 26 × 10 95.7 94.1 93.9 59674 68.5 65.9 62.1 3.4 × 108 88.0 83.5 76.8 38737 95.0 93.3 96.3 3.3 × 107 93.8 91.1 90.8 1.7 × 109 91.6 90.0 89.1 1.5 × 106 64.5 60.9 61.1 3392 90.7 92.6 93.7 1134750 80.3 78.2 77.0 4.3 × 107 86.44 84.49 82.65
#Sol 1.6 × 1019 3.2 × 1020 7.1 × 1014 3.1 × 1038 5.6 × 1011 2.1 × 109 1.2 × 1018 1.8 × 1018 1.2 × 1010 2.8 × 1024 4.6 × 1014 6.1 × 107 3.8 × 108 6.2 × 1014
does not seem to be improved by larger multi-trees. Nevertheless, the Occam hypothesis can also be regarded as a way to obtain more and more compact solutions without losing accuracy.
6
Conclusions
This work has presented a novel method for extracting a single solution from an ensemble of solutions without removing training data for validation. The most closely related work is Quinlan’s miniboosting [16]. However, Quinlan’s method can be considered an ensemble method which generates three trees, followed by a merging stage where a single but quite complex tree could be obtained. Moreover, as he recognises, “although it is (usually) possible to construct a single merged decision tree that induces the same partition as a small ensemble, the tree is so large that it conveys no insight. This is a pity, as insight was the prime motivation for producing a single tree”. We overcome the previous problem by producing a single tree that is based on a selection, using the combination as an oracle. Consequently, the result is a single comprehensible solution, an archetype or representative of the ensemble. As we have shown, the single solution obtained by our method is not 100% equivalent to the combination, but in general it gets reasonably close. On the other hand, it is clear that a similar technique could be used for regression models, using, e.g., minimum squared error as a discrepancy (similarity) metric. From a more general point of view, ensemble methods have been used as an argument in favor of the Epicurus criterion (all consistent models should be retained) and against Occam’s razor, because complex combined hypotheses usually obtain better results than the simplest solution [21]. A counterargument may be that the combination is usually expressed outside the hypothesis language. With our work, we have shown that even inside the hypothesis language, the shortest solution is not the best one.
176
C. Ferri, J. Hern´ andez-Orallo, and M.J. Ram´ırez-Quintana
With regard to future work, a mixture of the archetyping method and Occam’s razor could also be investigated. Another idea is that the oracle does not need to be an internal combined hypothesis but it can be any external source, such as a neural network. Therefore, this could be regarded as a new method to “convert” incomprehensible neural networks (or other models) to comprehensible models (with possibly a slight loss in accuracy), which could also be seen as an ‘explanation’ of the original model. Finally, it is important to clarify that an archetype solution cannot be obtained without an ensemble, and the quality of the representative would depend on the number of individual hypotheses in the ensemble. Note that this number is exponentially increased by the use of shared ensembles, in particular our multi-tree structure, without requiring a huge amount of memory. Nonetheless, a quite interesting open work would be to study specific methods to generate the ensemble (in our case, to construct the multi-tree) or to investigate the combination method that would produce better oracles for the selection of the archetype. Acknowledgements. We would like to thank the anonymous reviewers for suggesting the idea of using the archetype as an explanation of the ensemble.
References 1. Leo Breiman. Bagging predictors. Machine Learning, 24(2):123–140, 1996. 2. J.G. Cleary and L.E. Trigg. Experiences with ob1, an optimal bayes decision tree learner. Technical report, Department of Computer Science, Univ. of Waikato, New Zealand, 1998. 3. J. Cohen. A coefficient of agreement for nominal scales. Educational and Psychological Meas., 20:37–46, 1960. 4. T. G Dietterich. Ensemble methods in machine learning. In First International Workshop on Multiple Classifier Systems, pages 1–15, 2000. 5. Thomas G. Dietterich. An experimental comparison of three methods for constructing ensembles of decision trees: Bagging, Boosting, and Randomization. Machine Learning, 40(2):139–157, 2000. 6. C. Ferri, J. Hern´ andez, and M.J. Ram´ırez. Induction of Decision Multi-trees using Levin Search. In Int. Conf. on Computational Science, ICCS’02, LNCS, 2002. 7. C. Ferri, J. Hern´ andez, and M.J. Ram´ırez. Learning multiple and different hypotheses. Technical report, Department of Computer Science, Universitat Polit´ecnica de Val´encia, 2002. 8. C. Ferri, J. Hern´ andez, and M.J. Ram´ırez. SMILES system, a multi-purpose learning system. http://www.dsic.upv.es/˜flip/smiles/, 2002. 9. Y. Freund and R.E. Schapire. Experiments with a new boosting algorithm. In the 13th Int. Conf. on Machine Learning (ICML’1996), pages 148–156, 1996. 10. Tim Kam Ho. C4.5 decision forests. In Proc. of 14th Intl. Conf. on Pattern Recognition, Brisbane, Australia, pages 545–549, 1998. 11. Ludmila I. Kuncheva. A Theoretical Study on Six Classifier Fusion Strategies. IEEE Trans. on Pattern Analysis and Machine Intelligence, 24(2):281–286, 2002.
From Ensemble Methods to Comprehensible Models
177
12. Ludmila I. Kuncheva and Christopher J. Whitaker. Measures of diversity in classifier ensembles and their relationship with the ensemble accuracy. Submitted to Machine Learning, 2002. 13. Dragos D. Margineantu and Thomas G. Dietterich. Pruning adaptive boosting. In 14th Int. Conf. on Machine Learning, pages 211–218. Morgan Kaufmann, 1997. 14. N.J. Nilsson. Artificial Intelligence: a new synthesis. Morgan Kaufmann, 1998. 15. University of California. UCI Machine Learning Repository Content Summary. http://www.ics.uci.edu/˜mlearn/MLSummary.html. 16. J. Quinlan. Miniboosting decision trees. Submitted to JAIR, 1998. 17. J. R. Quinlan. C4.5: Programs for Machine Learning. Morgan Kaufmann, 1993. 18. J. R. Quinlan. Bagging, Boosting, and C4.5. In Proc. of the 13th Nat. Conf. on A.I. and the 8th Innovative Applications of A.I. Conf., pages 725–730. AAAI/MIT Press, 1996. 19. Ross Quinlan. Relational learning and boosting. In Saso Dzeroski and Nada Lavrac, editors, Relational Data Mining, pages 292–306. Springer-Verlag, September 2001. 20. P. Volf an